phase: 181_svg_document_preview
**Phase 181 — final verification pass (all tasks already complete; verified + docs pass)**
- Verified: `app/rag/svg_text.py` (100 % cov), vector branch in `_index_image_file`, `_row_kind_mismatch` text→image migration, `IMAGE_MIMES[".svg"]`, sandboxed CSP on `GET /api/documents/{id}/image`, `.doc-image-img--vector` in `document.js`/`styles.css`, figure/thumbnail deliberate non-changes.
- Tests/lint (exact commands): `uv run pytest --cov=app --cov-report=term-missing` → **4265 passed, TOTAL 99 %** (exit 0) · `uv run pytest tests/e2e/test_svg_document_preview.py -v --no-cov` → **5 passed in isolation** · `uv run ruff check .` → clean · `uv run pyright` → **0 errors**.
- Regression (each run in isolation, `--no-cov`, all rc=0): `test_image_documents`, `test_chat_image_questions`, `test_answer_images`, `test_chat_answer_clusters`, `test_image_anchor_context`, `test_image_upload_feedback`, `test_markdown_inline_links_images`, `test_prose_word_wrap`, `test_pdf_support`, `test_pictorial_pdf`, `test_garbage_pdf`, `test_pdf_image_extraction`, `test_pdf_image_context`, `test_pdf_pages_in_sources`, `test_pdf_strip_additive_selfheal`, `test_sync_image_progress`.
- Completion criteria: ①SVG→`is_image`+`image_path`, no vision call, own `<title>/<desc>/<text>` — **met**; ②`image/svg+xml` + `default-src 'none'; sandbox` + `private, max-age=3600`, raster headers byte-unchanged — **met**; ③chat figure → modal `<img>`, no XML — **met**; ④`/document.html` full-width vector in `.doc-image` + text below — **met**; ⑤`` paints — **met**; ⑥raster/PDF unchanged — **met**; ⑦all gates green — **met**; ⑧commit + phase-dir move — harness-owned.
- Documentation: found and fixed README inconsistencies — stale forward reference "left to phase 181 / is phase 181's job" → completed-tense sizing note; "Updating the documents" standalone-image list missing `svg`; new **Raster only** bullet for question uploads (`.svg` → 422) documenting the `chat_images` guard; missing `svg_indexed=N` / `import: svg indexed` log contract; env-table heading `122/123/124` → `122/123/124/181`; new troubleshooting entry for an `.svg` showing raw XML. Full audit: every `Settings` field and every `.env.example` var is documented; pages/commands/scripts all exist. AGENTS.md verified accurate (phase-181 §5 clause present) — no changes needed beyond what the phase wrote.
- Notable deviation (kept from task 02): `app/api/chat_images.py` subtracts `_VECTOR_IMAGE_EXTENSIONS` from the question-image accept set — a required consequence of LOCKED A7 (`svg` in the default image set) since `_CHAT_IMAGE_FILENAME_RE` refuses to *serve* `.svg`; the regex itself is untouched, as the phase specified.
- Note for the commit: changes include `README.md`, `AGENTS.md` and `.env.example` — the phase file's `git add app/ frontend/ tests/ .env.example .agents/phases/` line omits the two markdown files; stage them too.
- Next pending phase: `182_verbatim_show_text`.
This commit is contained in:
@@ -0,0 +1,10 @@
|
|||||||
|
**Phase 181 — final verification pass (all tasks already complete; verified + docs pass)**
|
||||||
|
|
||||||
|
- Verified: `app/rag/svg_text.py` (100 % cov), vector branch in `_index_image_file`, `_row_kind_mismatch` text→image migration, `IMAGE_MIMES[".svg"]`, sandboxed CSP on `GET /api/documents/{id}/image`, `.doc-image-img--vector` in `document.js`/`styles.css`, figure/thumbnail deliberate non-changes.
|
||||||
|
- Tests/lint (exact commands): `uv run pytest --cov=app --cov-report=term-missing` → **4265 passed, TOTAL 99 %** (exit 0) · `uv run pytest tests/e2e/test_svg_document_preview.py -v --no-cov` → **5 passed in isolation** · `uv run ruff check .` → clean · `uv run pyright` → **0 errors**.
|
||||||
|
- Regression (each run in isolation, `--no-cov`, all rc=0): `test_image_documents`, `test_chat_image_questions`, `test_answer_images`, `test_chat_answer_clusters`, `test_image_anchor_context`, `test_image_upload_feedback`, `test_markdown_inline_links_images`, `test_prose_word_wrap`, `test_pdf_support`, `test_pictorial_pdf`, `test_garbage_pdf`, `test_pdf_image_extraction`, `test_pdf_image_context`, `test_pdf_pages_in_sources`, `test_pdf_strip_additive_selfheal`, `test_sync_image_progress`.
|
||||||
|
- Completion criteria: ①SVG→`is_image`+`image_path`, no vision call, own `<title>/<desc>/<text>` — **met**; ②`image/svg+xml` + `default-src 'none'; sandbox` + `private, max-age=3600`, raster headers byte-unchanged — **met**; ③chat figure → modal `<img>`, no XML — **met**; ④`/document.html` full-width vector in `.doc-image` + text below — **met**; ⑤`` paints — **met**; ⑥raster/PDF unchanged — **met**; ⑦all gates green — **met**; ⑧commit + phase-dir move — harness-owned.
|
||||||
|
- Documentation: found and fixed README inconsistencies — stale forward reference "left to phase 181 / is phase 181's job" → completed-tense sizing note; "Updating the documents" standalone-image list missing `svg`; new **Raster only** bullet for question uploads (`.svg` → 422) documenting the `chat_images` guard; missing `svg_indexed=N` / `import: svg indexed` log contract; env-table heading `122/123/124` → `122/123/124/181`; new troubleshooting entry for an `.svg` showing raw XML. Full audit: every `Settings` field and every `.env.example` var is documented; pages/commands/scripts all exist. AGENTS.md verified accurate (phase-181 §5 clause present) — no changes needed beyond what the phase wrote.
|
||||||
|
- Notable deviation (kept from task 02): `app/api/chat_images.py` subtracts `_VECTOR_IMAGE_EXTENSIONS` from the question-image accept set — a required consequence of LOCKED A7 (`svg` in the default image set) since `_CHAT_IMAGE_FILENAME_RE` refuses to *serve* `.svg`; the regex itself is untouched, as the phase specified.
|
||||||
|
- Note for the commit: changes include `README.md`, `AGENTS.md` and `.env.example` — the phase file's `git add app/ frontend/ tests/ .env.example .agents/phases/` line omits the two markdown files; stage them too.
|
||||||
|
- Next pending phase: `182_verbatim_show_text`.
|
||||||
+1
@@ -0,0 +1 @@
|
|||||||
|
ok
|
||||||
+139
@@ -0,0 +1,139 @@
|
|||||||
|
........................................................................ [ 1%]
|
||||||
|
........................................................................ [ 3%]
|
||||||
|
........................................................................ [ 5%]
|
||||||
|
........................................................................ [ 6%]
|
||||||
|
........................................................................ [ 8%]
|
||||||
|
........................................................................ [ 10%]
|
||||||
|
........................................................................ [ 11%]
|
||||||
|
........................................................................ [ 13%]
|
||||||
|
........................................................................ [ 15%]
|
||||||
|
........................................................................ [ 16%]
|
||||||
|
........................................................................ [ 18%]
|
||||||
|
........................................................................ [ 20%]
|
||||||
|
........................................................................ [ 21%]
|
||||||
|
........................................................................ [ 23%]
|
||||||
|
........................................................................ [ 25%]
|
||||||
|
........................................................................ [ 27%]
|
||||||
|
........................................................................ [ 28%]
|
||||||
|
........................................................................ [ 30%]
|
||||||
|
........................................................................ [ 32%]
|
||||||
|
........................................................................ [ 33%]
|
||||||
|
........................................................................ [ 35%]
|
||||||
|
........................................................................ [ 37%]
|
||||||
|
........................................................................ [ 38%]
|
||||||
|
........................................................................ [ 40%]
|
||||||
|
........................................................................ [ 42%]
|
||||||
|
........................................................................ [ 43%]
|
||||||
|
........................................................................ [ 45%]
|
||||||
|
........................................................................ [ 47%]
|
||||||
|
........................................................................ [ 48%]
|
||||||
|
........................................................................ [ 50%]
|
||||||
|
........................................................................ [ 52%]
|
||||||
|
........................................................................ [ 54%]
|
||||||
|
........................................................................ [ 55%]
|
||||||
|
........................................................................ [ 57%]
|
||||||
|
........................................................................ [ 59%]
|
||||||
|
........................................................................ [ 60%]
|
||||||
|
........................................................................ [ 62%]
|
||||||
|
........................................................................ [ 64%]
|
||||||
|
........................................................................ [ 65%]
|
||||||
|
........................................................................ [ 67%]
|
||||||
|
........................................................................ [ 69%]
|
||||||
|
........................................................................ [ 70%]
|
||||||
|
........................................................................ [ 72%]
|
||||||
|
........................................................................ [ 74%]
|
||||||
|
........................................................................ [ 75%]
|
||||||
|
........................................................................ [ 77%]
|
||||||
|
........................................................................ [ 79%]
|
||||||
|
........................................................................ [ 81%]
|
||||||
|
........................................................................ [ 82%]
|
||||||
|
........................................................................ [ 84%]
|
||||||
|
........................................................................ [ 86%]
|
||||||
|
........................................................................ [ 87%]
|
||||||
|
........................................................................ [ 89%]
|
||||||
|
........................................................................ [ 91%]
|
||||||
|
........................................................................ [ 92%]
|
||||||
|
........................................................................ [ 94%]
|
||||||
|
........................................................................ [ 96%]
|
||||||
|
........................................................................ [ 97%]
|
||||||
|
........................................................................ [ 99%]
|
||||||
|
................. [100%]
|
||||||
|
=============================== warnings summary ===============================
|
||||||
|
.venv/lib/python3.13/site-packages/fastapi/testclient.py:1
|
||||||
|
/var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
|
||||||
|
from starlette.testclient import TestClient as TestClient # noqa
|
||||||
|
|
||||||
|
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
|
||||||
|
================================ tests coverage ================================
|
||||||
|
_______________ coverage: platform linux, python 3.13.13-final-0 _______________
|
||||||
|
|
||||||
|
Name Stmts Miss Cover
|
||||||
|
--------------------------------------------------
|
||||||
|
app/__init__.py 1 0 100%
|
||||||
|
app/api/__init__.py 0 0 100%
|
||||||
|
app/api/admin_settings.py 114 0 100%
|
||||||
|
app/api/auth.py 52 0 100%
|
||||||
|
app/api/chat.py 280 1 99%
|
||||||
|
app/api/chat_images.py 52 0 100%
|
||||||
|
app/api/chats.py 111 0 100%
|
||||||
|
app/api/config.py 13 0 100%
|
||||||
|
app/api/doc_drafts.py 99 0 100%
|
||||||
|
app/api/docs.py 239 1 99%
|
||||||
|
app/api/git_sources.py 259 0 100%
|
||||||
|
app/api/health.py 10 0 100%
|
||||||
|
app/api/steering.py 42 0 100%
|
||||||
|
app/api/suggestions.py 33 0 100%
|
||||||
|
app/api/sync.py 204 0 100%
|
||||||
|
app/api/tokens.py 40 0 100%
|
||||||
|
app/api/ui_settings.py 55 0 100%
|
||||||
|
app/config.py 289 0 100%
|
||||||
|
app/core/__init__.py 0 0 100%
|
||||||
|
app/core/auth.py 45 0 100%
|
||||||
|
app/core/caching.py 124 0 100%
|
||||||
|
app/core/compression.py 59 0 100%
|
||||||
|
app/core/debugging.py 29 2 93%
|
||||||
|
app/core/docs_push.py 43 0 100%
|
||||||
|
app/core/errors.py 5 0 100%
|
||||||
|
app/core/http_cache.py 22 0 100%
|
||||||
|
app/core/logging.py 13 0 100%
|
||||||
|
app/core/rate_limit.py 44 0 100%
|
||||||
|
app/core/response_budget.py 63 0 100%
|
||||||
|
app/core/security_headers.py 20 0 100%
|
||||||
|
app/core/settings_catalog.py 18 0 100%
|
||||||
|
app/core/settings_migrate.py 32 0 100%
|
||||||
|
app/core/settings_overlay.py 111 0 100%
|
||||||
|
app/core/theming.py 40 0 100%
|
||||||
|
app/core/tokens.py 44 0 100%
|
||||||
|
app/db.py 22 0 100%
|
||||||
|
app/main.py 105 0 100%
|
||||||
|
app/models.py 141 0 100%
|
||||||
|
app/rag/__init__.py 0 0 100%
|
||||||
|
app/rag/agent.py 454 1 99%
|
||||||
|
app/rag/archive_upload.py 134 0 100%
|
||||||
|
app/rag/chunker.py 220 4 98%
|
||||||
|
app/rag/doc_dates.py 18 0 100%
|
||||||
|
app/rag/folder_summaries.py 134 0 100%
|
||||||
|
app/rag/git_sources.py 77 0 100%
|
||||||
|
app/rag/importer.py 855 4 99%
|
||||||
|
app/rag/llm.py 253 1 99%
|
||||||
|
app/rag/md_images.py 63 0 100%
|
||||||
|
app/rag/overview.py 78 0 100%
|
||||||
|
app/rag/pdf.py 192 0 100%
|
||||||
|
app/rag/prompts.py 156 0 100%
|
||||||
|
app/rag/retriever.py 252 3 99%
|
||||||
|
app/rag/scaffolding.py 55 0 100%
|
||||||
|
app/rag/source_removal.py 41 0 100%
|
||||||
|
app/rag/sources_meta.py 16 0 100%
|
||||||
|
app/rag/suggestions.py 27 0 100%
|
||||||
|
app/rag/summarizer.py 58 0 100%
|
||||||
|
app/rag/svg_text.py 111 0 100%
|
||||||
|
app/schemas.py 533 0 100%
|
||||||
|
--------------------------------------------------
|
||||||
|
TOTAL 6600 17 99%
|
||||||
|
coverage gate: app/ 99% (>90%) OK
|
||||||
|
All checks passed!
|
||||||
|
0 errors, 0 warnings, 0 informations
|
||||||
|
WARNING: there is a new pyright version available (v1.1.411 -> v1.1.414).
|
||||||
|
Please install the new version or set PYRIGHT_PYTHON_FORCE_VERSION to `latest`
|
||||||
|
|
||||||
|
validation OK
|
||||||
+18
@@ -0,0 +1,18 @@
|
|||||||
|
**Implemented** (task 01 — repo-side only, no live infra touched):
|
||||||
|
- `app/rag/svg_text.py` (new, pure, stdlib-only, `md_images.py` shape): `MAX_SVG_BYTES=2 MiB`, `MAX_SVG_TEXT_CHARS=4000`, frozen `SvgText(text, structural)`, `extract_svg_text(data, *, stem="")`, `local_name(tag)`.
|
||||||
|
- Pre-parse guards (LOCKED A3 sibling): empty/whitespace-only, `>MAX_SVG_BYTES`, case-insensitive `<!doctype`/`<!entity` refusal (stdlib otherwise *expands* internal entities — pinned by test), `ParseError` → `None` (never raises).
|
||||||
|
- Extraction (D2): `title`/`desc` (concat) + `text` (space-joined, covers `<tspan>`) in **document order** by local name (Clark-notation / declared-prefix agnostic), whitespace-normalised, case-insensitive dedupe keeping first, `\n`-joined, entity-decoded, capped at the last newline/period before 4000 (hard cut only if no boundary).
|
||||||
|
- LOCKED A2 fallback: `Vector diagram "<stem>" — <N> shapes (<top-3 kinds>), viewBox <w>×<h>`; every clause omitted when unstated, `1 shape`/`2 shapes` plural, top-3 by count with document-order tie-break, viewBox requires 4 finite numbers, integral values print without `.0`, empty stem → no empty quotes.
|
||||||
|
- `tests/unit/test_svg_text_extraction.py` — 45 pure-function tests (guards, order, tspans, namespaces, `xlink:` ignored, dedupe, entities, cap variants, all fallback clause/plural/omission shapes, comments, non-`svg` root).
|
||||||
|
|
||||||
|
**Gates (all 100% green):**
|
||||||
|
- `uv run pytest tests/unit/test_svg_text_extraction.py -q` → 45 passed; `--cov=app.rag.svg_text` → **100%**
|
||||||
|
- `uv run pytest --cov=app --cov-report=term-missing` → **4243 passed**, TOTAL **99%** (svg_text 111/111)
|
||||||
|
- `uv run ruff check .` → All checks passed; `uv run pyright` → 0 errors
|
||||||
|
- Completion-criteria one-liners verified: `SvgText(text='Top\na', structural=False)`, `Vector diagram "topo" — 1 shape (path)`, DOCTYPE case → `None`.
|
||||||
|
|
||||||
|
**Docs:** no doc impact — the module is not yet wired (no user-visible surface); README/AGENTS updates belong to tasks 02–04.
|
||||||
|
|
||||||
|
**Decisions:** (1) per-kind count in the fallback listing is shown only when it is >1 (`3 shapes (path×3)` vs `1 shape (path)`), so the task's pinned literal holds while D3's "with counts" still reads off multi-kind files; (2) added a non-finite guard (`NaN`/`inf` viewBox → clause omitted) so the line never states a size the file doesn't; (3) viewBox clause sits after the shape clause joined by `, `, and uses ` — ` as the single separator when shapes are absent (`Vector diagram "canvas" — viewBox 16×9`).
|
||||||
|
|
||||||
|
**Next pending task:** `.agents/phases/todo/181_svg_document_preview/02_svg_index_as_image.md`
|
||||||
+1
@@ -0,0 +1 @@
|
|||||||
|
ok
|
||||||
+138
@@ -0,0 +1,138 @@
|
|||||||
|
........................................................................ [ 1%]
|
||||||
|
........................................................................ [ 3%]
|
||||||
|
........................................................................ [ 5%]
|
||||||
|
........................................................................ [ 6%]
|
||||||
|
........................................................................ [ 8%]
|
||||||
|
........................................................................ [ 10%]
|
||||||
|
........................................................................ [ 11%]
|
||||||
|
........................................................................ [ 13%]
|
||||||
|
........................................................................ [ 15%]
|
||||||
|
........................................................................ [ 16%]
|
||||||
|
........................................................................ [ 18%]
|
||||||
|
........................................................................ [ 20%]
|
||||||
|
........................................................................ [ 22%]
|
||||||
|
........................................................................ [ 23%]
|
||||||
|
........................................................................ [ 25%]
|
||||||
|
........................................................................ [ 27%]
|
||||||
|
........................................................................ [ 28%]
|
||||||
|
........................................................................ [ 30%]
|
||||||
|
........................................................................ [ 32%]
|
||||||
|
........................................................................ [ 33%]
|
||||||
|
........................................................................ [ 35%]
|
||||||
|
........................................................................ [ 37%]
|
||||||
|
........................................................................ [ 39%]
|
||||||
|
........................................................................ [ 40%]
|
||||||
|
........................................................................ [ 42%]
|
||||||
|
........................................................................ [ 44%]
|
||||||
|
........................................................................ [ 45%]
|
||||||
|
........................................................................ [ 47%]
|
||||||
|
........................................................................ [ 49%]
|
||||||
|
........................................................................ [ 50%]
|
||||||
|
........................................................................ [ 52%]
|
||||||
|
........................................................................ [ 54%]
|
||||||
|
........................................................................ [ 55%]
|
||||||
|
........................................................................ [ 57%]
|
||||||
|
........................................................................ [ 59%]
|
||||||
|
........................................................................ [ 61%]
|
||||||
|
........................................................................ [ 62%]
|
||||||
|
........................................................................ [ 64%]
|
||||||
|
........................................................................ [ 66%]
|
||||||
|
........................................................................ [ 67%]
|
||||||
|
........................................................................ [ 69%]
|
||||||
|
........................................................................ [ 71%]
|
||||||
|
........................................................................ [ 72%]
|
||||||
|
........................................................................ [ 74%]
|
||||||
|
........................................................................ [ 76%]
|
||||||
|
........................................................................ [ 78%]
|
||||||
|
........................................................................ [ 79%]
|
||||||
|
........................................................................ [ 81%]
|
||||||
|
........................................................................ [ 83%]
|
||||||
|
........................................................................ [ 84%]
|
||||||
|
........................................................................ [ 86%]
|
||||||
|
........................................................................ [ 88%]
|
||||||
|
........................................................................ [ 89%]
|
||||||
|
........................................................................ [ 91%]
|
||||||
|
........................................................................ [ 93%]
|
||||||
|
........................................................................ [ 95%]
|
||||||
|
........................................................................ [ 96%]
|
||||||
|
........................................................................ [ 98%]
|
||||||
|
................................................................... [100%]
|
||||||
|
=============================== warnings summary ===============================
|
||||||
|
.venv/lib/python3.13/site-packages/fastapi/testclient.py:1
|
||||||
|
/var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
|
||||||
|
from starlette.testclient import TestClient as TestClient # noqa
|
||||||
|
|
||||||
|
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
|
||||||
|
================================ tests coverage ================================
|
||||||
|
_______________ coverage: platform linux, python 3.13.13-final-0 _______________
|
||||||
|
|
||||||
|
Name Stmts Miss Cover
|
||||||
|
--------------------------------------------------
|
||||||
|
app/__init__.py 1 0 100%
|
||||||
|
app/api/__init__.py 0 0 100%
|
||||||
|
app/api/admin_settings.py 114 0 100%
|
||||||
|
app/api/auth.py 52 0 100%
|
||||||
|
app/api/chat.py 280 1 99%
|
||||||
|
app/api/chat_images.py 50 0 100%
|
||||||
|
app/api/chats.py 111 0 100%
|
||||||
|
app/api/config.py 13 0 100%
|
||||||
|
app/api/doc_drafts.py 99 0 100%
|
||||||
|
app/api/docs.py 234 1 99%
|
||||||
|
app/api/git_sources.py 259 0 100%
|
||||||
|
app/api/health.py 10 0 100%
|
||||||
|
app/api/steering.py 42 0 100%
|
||||||
|
app/api/suggestions.py 33 0 100%
|
||||||
|
app/api/sync.py 204 0 100%
|
||||||
|
app/api/tokens.py 40 0 100%
|
||||||
|
app/api/ui_settings.py 55 0 100%
|
||||||
|
app/config.py 289 0 100%
|
||||||
|
app/core/__init__.py 0 0 100%
|
||||||
|
app/core/auth.py 45 0 100%
|
||||||
|
app/core/caching.py 124 0 100%
|
||||||
|
app/core/compression.py 59 0 100%
|
||||||
|
app/core/debugging.py 29 2 93%
|
||||||
|
app/core/docs_push.py 43 0 100%
|
||||||
|
app/core/errors.py 5 0 100%
|
||||||
|
app/core/http_cache.py 22 0 100%
|
||||||
|
app/core/logging.py 13 0 100%
|
||||||
|
app/core/rate_limit.py 44 0 100%
|
||||||
|
app/core/response_budget.py 63 0 100%
|
||||||
|
app/core/security_headers.py 20 0 100%
|
||||||
|
app/core/settings_catalog.py 18 0 100%
|
||||||
|
app/core/settings_migrate.py 32 0 100%
|
||||||
|
app/core/settings_overlay.py 111 0 100%
|
||||||
|
app/core/theming.py 40 0 100%
|
||||||
|
app/core/tokens.py 44 0 100%
|
||||||
|
app/db.py 22 0 100%
|
||||||
|
app/main.py 105 0 100%
|
||||||
|
app/models.py 141 0 100%
|
||||||
|
app/rag/__init__.py 0 0 100%
|
||||||
|
app/rag/agent.py 454 1 99%
|
||||||
|
app/rag/archive_upload.py 134 0 100%
|
||||||
|
app/rag/chunker.py 220 4 98%
|
||||||
|
app/rag/doc_dates.py 18 0 100%
|
||||||
|
app/rag/folder_summaries.py 134 0 100%
|
||||||
|
app/rag/git_sources.py 77 0 100%
|
||||||
|
app/rag/importer.py 841 4 99%
|
||||||
|
app/rag/llm.py 253 1 99%
|
||||||
|
app/rag/md_images.py 63 0 100%
|
||||||
|
app/rag/overview.py 78 0 100%
|
||||||
|
app/rag/pdf.py 192 0 100%
|
||||||
|
app/rag/prompts.py 156 0 100%
|
||||||
|
app/rag/retriever.py 252 3 99%
|
||||||
|
app/rag/scaffolding.py 55 0 100%
|
||||||
|
app/rag/source_removal.py 41 0 100%
|
||||||
|
app/rag/sources_meta.py 16 0 100%
|
||||||
|
app/rag/suggestions.py 27 0 100%
|
||||||
|
app/rag/summarizer.py 58 0 100%
|
||||||
|
app/rag/svg_text.py 111 0 100%
|
||||||
|
app/schemas.py 533 0 100%
|
||||||
|
--------------------------------------------------
|
||||||
|
TOTAL 6579 17 99%
|
||||||
|
coverage gate: app/ 99% (>90%) OK
|
||||||
|
All checks passed!
|
||||||
|
0 errors, 0 warnings, 0 informations
|
||||||
|
WARNING: there is a new pyright version available (v1.1.411 -> v1.1.414).
|
||||||
|
Please install the new version or set PYRIGHT_PYTHON_FORCE_VERSION to `latest`
|
||||||
|
|
||||||
|
validation OK
|
||||||
+24
@@ -0,0 +1,24 @@
|
|||||||
|
All checks green. Report:
|
||||||
|
|
||||||
|
**Implemented** (task 02; work was largely staged by earlier context-killed attempts — I verified every step against the task file, ran the gates, and confirmed no gaps):
|
||||||
|
- `app/config.py`: `image_extensions` default → `png,jpg,jpeg,webp,gif,bmp,svg` + doc-comment on SVG as the one VECTOR member (A1) and `BOR_IMAGES` as the single knob (A7).
|
||||||
|
- `app/rag/importer.py`: vector branch in `_index_image_file` (`extract_svg_text` replaces `_describe_or_skip` for `.svg`, everything else shared), `svg_indexed` log-only counter + `import: svg indexed … structural=… chars=…` line (AGENTS rule 10), module/branch docstrings, and the D4 fast-path guard `_row_kind_mismatch()` (text-SVG rows convert on the next sync; toggle-off direction untouched per A7).
|
||||||
|
- Necessary companion: the image branch's own content-digest skip now also requires `doc.is_image` (an ASCII file's text digest equals its byte digest — without it the migration would report `unchanged` forever); documented + pinned by test.
|
||||||
|
- `app/api/chat_images.py`: question uploads = `image_extension_set − _VECTOR_IMAGE_EXTENSIONS` (`.svg` 422s — a vision endpoint rejects `image/svg+xml` and the serve route's regex would never serve it); `_CHAT_IMAGE_FILENAME_RE` left untouched per phase "NOT touched".
|
||||||
|
- Docs: `.env.example` (+ vector/BOR_IMAGES=0/paperclip notes), `README.md` formats paragraph, new "SVG is the one vector member" paragraph, settings-table row.
|
||||||
|
- Tests: new `tests/unit/test_svg_image_ingest.py` (10 tests: no-vision-call proof via sentinel seam, content/summary/chunks/copy/`svg_indexed`, structural-line textless SVG, over-cap + malformed fail-soft (no row, no copy), phase-125 `image_context` prepend, text→image conversion then fast path resumes, raster/vector fast-path regression, `_row_kind_mismatch` truth table); updated `test_image_documents.py`, `test_importer.py` (log line), `test_chat_image_questions.py`.
|
||||||
|
|
||||||
|
**Gates**
|
||||||
|
- `uv run pytest --cov=app --cov-report=term-missing` → **4253 passed**, TOTAL **99%** (`app/rag/svg_text.py` 100%, `app/rag/importer.py` 99% — only pre-existing misses).
|
||||||
|
- `uv run ruff check .` → All checks passed; `uv run pyright` → 0 errors/0 warnings.
|
||||||
|
- Targeted: `pytest tests/unit/test_svg_*.py tests/unit/test_image_documents.py tests/unit/test_chat_image_questions.py tests/integration/test_importer{,_dates}.py tests/integration/test_import_docs_git.py --no-cov` → 240 passed.
|
||||||
|
- E2E regression (image surfaces touched by the default change): `tests/e2e/test_image_documents.py` → 4 passed; `test_chat_image_questions.py` + `test_answer_images.py` → 10 passed.
|
||||||
|
|
||||||
|
**Docs**: `README.md` + `.env.example` updated as above; `AGENTS.md` no doc impact (no extension list there).
|
||||||
|
|
||||||
|
**Decisions / deviations**
|
||||||
|
1. Added `doc.is_image` to `_index_image_file`'s unchanged guard (beyond "keep everything after the seam as-is") — required for D4 to work; explained inline and pinned by `test_text_indexed_svg_row_is_converted_on_the_next_sync`.
|
||||||
|
2. Fail-soft log now reads `import: svg text extraction failed` for the vector case (raster wording byte-identical) for auditability; counter/behaviour unchanged.
|
||||||
|
3. **Owner action (I did not touch the user-owned live `.env`)**: `.env:63` still pins `BOR_IMAGE_EXTENSIONS=png,jpg,jpeg,webp,gif,bmp`, so the new default has no effect on the running app until `svg` is added (or the line removed) — LOCKED A7 / phase says a live `.env` keeps its value.
|
||||||
|
|
||||||
|
**Next pending task**: `.agents/phases/todo/181_svg_document_preview/03_svg_bytes_route_csp.md`
|
||||||
+1
@@ -0,0 +1 @@
|
|||||||
|
ok
|
||||||
+139
@@ -0,0 +1,139 @@
|
|||||||
|
........................................................................ [ 1%]
|
||||||
|
........................................................................ [ 3%]
|
||||||
|
........................................................................ [ 5%]
|
||||||
|
........................................................................ [ 6%]
|
||||||
|
........................................................................ [ 8%]
|
||||||
|
........................................................................ [ 10%]
|
||||||
|
........................................................................ [ 11%]
|
||||||
|
........................................................................ [ 13%]
|
||||||
|
........................................................................ [ 15%]
|
||||||
|
........................................................................ [ 16%]
|
||||||
|
........................................................................ [ 18%]
|
||||||
|
........................................................................ [ 20%]
|
||||||
|
........................................................................ [ 22%]
|
||||||
|
........................................................................ [ 23%]
|
||||||
|
........................................................................ [ 25%]
|
||||||
|
........................................................................ [ 27%]
|
||||||
|
........................................................................ [ 28%]
|
||||||
|
........................................................................ [ 30%]
|
||||||
|
........................................................................ [ 32%]
|
||||||
|
........................................................................ [ 33%]
|
||||||
|
........................................................................ [ 35%]
|
||||||
|
........................................................................ [ 37%]
|
||||||
|
........................................................................ [ 38%]
|
||||||
|
........................................................................ [ 40%]
|
||||||
|
........................................................................ [ 42%]
|
||||||
|
........................................................................ [ 44%]
|
||||||
|
........................................................................ [ 45%]
|
||||||
|
........................................................................ [ 47%]
|
||||||
|
........................................................................ [ 49%]
|
||||||
|
........................................................................ [ 50%]
|
||||||
|
........................................................................ [ 52%]
|
||||||
|
........................................................................ [ 54%]
|
||||||
|
........................................................................ [ 55%]
|
||||||
|
........................................................................ [ 57%]
|
||||||
|
........................................................................ [ 59%]
|
||||||
|
........................................................................ [ 60%]
|
||||||
|
........................................................................ [ 62%]
|
||||||
|
........................................................................ [ 64%]
|
||||||
|
........................................................................ [ 66%]
|
||||||
|
........................................................................ [ 67%]
|
||||||
|
........................................................................ [ 69%]
|
||||||
|
........................................................................ [ 71%]
|
||||||
|
........................................................................ [ 72%]
|
||||||
|
........................................................................ [ 74%]
|
||||||
|
........................................................................ [ 76%]
|
||||||
|
........................................................................ [ 77%]
|
||||||
|
........................................................................ [ 79%]
|
||||||
|
........................................................................ [ 81%]
|
||||||
|
........................................................................ [ 82%]
|
||||||
|
........................................................................ [ 84%]
|
||||||
|
........................................................................ [ 86%]
|
||||||
|
........................................................................ [ 88%]
|
||||||
|
........................................................................ [ 89%]
|
||||||
|
........................................................................ [ 91%]
|
||||||
|
........................................................................ [ 93%]
|
||||||
|
........................................................................ [ 94%]
|
||||||
|
........................................................................ [ 96%]
|
||||||
|
........................................................................ [ 98%]
|
||||||
|
........................................................................ [ 99%]
|
||||||
|
..... [100%]
|
||||||
|
=============================== warnings summary ===============================
|
||||||
|
.venv/lib/python3.13/site-packages/fastapi/testclient.py:1
|
||||||
|
/var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
|
||||||
|
from starlette.testclient import TestClient as TestClient # noqa
|
||||||
|
|
||||||
|
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
|
||||||
|
================================ tests coverage ================================
|
||||||
|
_______________ coverage: platform linux, python 3.13.13-final-0 _______________
|
||||||
|
|
||||||
|
Name Stmts Miss Cover
|
||||||
|
--------------------------------------------------
|
||||||
|
app/__init__.py 1 0 100%
|
||||||
|
app/api/__init__.py 0 0 100%
|
||||||
|
app/api/admin_settings.py 114 0 100%
|
||||||
|
app/api/auth.py 52 0 100%
|
||||||
|
app/api/chat.py 280 1 99%
|
||||||
|
app/api/chat_images.py 52 0 100%
|
||||||
|
app/api/chats.py 111 0 100%
|
||||||
|
app/api/config.py 13 0 100%
|
||||||
|
app/api/doc_drafts.py 99 0 100%
|
||||||
|
app/api/docs.py 234 1 99%
|
||||||
|
app/api/git_sources.py 259 0 100%
|
||||||
|
app/api/health.py 10 0 100%
|
||||||
|
app/api/steering.py 42 0 100%
|
||||||
|
app/api/suggestions.py 33 0 100%
|
||||||
|
app/api/sync.py 204 0 100%
|
||||||
|
app/api/tokens.py 40 0 100%
|
||||||
|
app/api/ui_settings.py 55 0 100%
|
||||||
|
app/config.py 289 0 100%
|
||||||
|
app/core/__init__.py 0 0 100%
|
||||||
|
app/core/auth.py 45 0 100%
|
||||||
|
app/core/caching.py 124 0 100%
|
||||||
|
app/core/compression.py 59 0 100%
|
||||||
|
app/core/debugging.py 29 2 93%
|
||||||
|
app/core/docs_push.py 43 0 100%
|
||||||
|
app/core/errors.py 5 0 100%
|
||||||
|
app/core/http_cache.py 22 0 100%
|
||||||
|
app/core/logging.py 13 0 100%
|
||||||
|
app/core/rate_limit.py 44 0 100%
|
||||||
|
app/core/response_budget.py 63 0 100%
|
||||||
|
app/core/security_headers.py 20 0 100%
|
||||||
|
app/core/settings_catalog.py 18 0 100%
|
||||||
|
app/core/settings_migrate.py 32 0 100%
|
||||||
|
app/core/settings_overlay.py 111 0 100%
|
||||||
|
app/core/theming.py 40 0 100%
|
||||||
|
app/core/tokens.py 44 0 100%
|
||||||
|
app/db.py 22 0 100%
|
||||||
|
app/main.py 105 0 100%
|
||||||
|
app/models.py 141 0 100%
|
||||||
|
app/rag/__init__.py 0 0 100%
|
||||||
|
app/rag/agent.py 454 1 99%
|
||||||
|
app/rag/archive_upload.py 134 0 100%
|
||||||
|
app/rag/chunker.py 220 4 98%
|
||||||
|
app/rag/doc_dates.py 18 0 100%
|
||||||
|
app/rag/folder_summaries.py 134 0 100%
|
||||||
|
app/rag/git_sources.py 77 0 100%
|
||||||
|
app/rag/importer.py 855 4 99%
|
||||||
|
app/rag/llm.py 253 1 99%
|
||||||
|
app/rag/md_images.py 63 0 100%
|
||||||
|
app/rag/overview.py 78 0 100%
|
||||||
|
app/rag/pdf.py 192 0 100%
|
||||||
|
app/rag/prompts.py 156 0 100%
|
||||||
|
app/rag/retriever.py 252 3 99%
|
||||||
|
app/rag/scaffolding.py 55 0 100%
|
||||||
|
app/rag/source_removal.py 41 0 100%
|
||||||
|
app/rag/sources_meta.py 16 0 100%
|
||||||
|
app/rag/suggestions.py 27 0 100%
|
||||||
|
app/rag/summarizer.py 58 0 100%
|
||||||
|
app/rag/svg_text.py 111 0 100%
|
||||||
|
app/schemas.py 533 0 100%
|
||||||
|
--------------------------------------------------
|
||||||
|
TOTAL 6595 17 99%
|
||||||
|
coverage gate: app/ 99% (>90%) OK
|
||||||
|
All checks passed!
|
||||||
|
0 errors, 0 warnings, 0 informations
|
||||||
|
WARNING: there is a new pyright version available (v1.1.411 -> v1.1.414).
|
||||||
|
Please install the new version or set PYRIGHT_PYTHON_FORCE_VERSION to `latest`
|
||||||
|
|
||||||
|
validation OK
|
||||||
+1
@@ -0,0 +1 @@
|
|||||||
|
(no final assistant message — see the .err log)
|
||||||
+1
@@ -0,0 +1 @@
|
|||||||
|
error: assistant error: 400: {"message":"litellm.BadRequestError: OpenAIException - prompt exceeds the 131072-token context; increase --context or shorten the convers…
|
||||||
+139
@@ -0,0 +1,139 @@
|
|||||||
|
........................................................................ [ 1%]
|
||||||
|
........................................................................ [ 3%]
|
||||||
|
........................................................................ [ 5%]
|
||||||
|
........................................................................ [ 6%]
|
||||||
|
........................................................................ [ 8%]
|
||||||
|
........................................................................ [ 10%]
|
||||||
|
........................................................................ [ 11%]
|
||||||
|
........................................................................ [ 13%]
|
||||||
|
........................................................................ [ 15%]
|
||||||
|
........................................................................ [ 16%]
|
||||||
|
........................................................................ [ 18%]
|
||||||
|
........................................................................ [ 20%]
|
||||||
|
........................................................................ [ 22%]
|
||||||
|
........................................................................ [ 23%]
|
||||||
|
........................................................................ [ 25%]
|
||||||
|
........................................................................ [ 27%]
|
||||||
|
........................................................................ [ 28%]
|
||||||
|
........................................................................ [ 30%]
|
||||||
|
........................................................................ [ 32%]
|
||||||
|
........................................................................ [ 33%]
|
||||||
|
........................................................................ [ 35%]
|
||||||
|
........................................................................ [ 37%]
|
||||||
|
........................................................................ [ 38%]
|
||||||
|
........................................................................ [ 40%]
|
||||||
|
........................................................................ [ 42%]
|
||||||
|
........................................................................ [ 44%]
|
||||||
|
........................................................................ [ 45%]
|
||||||
|
........................................................................ [ 47%]
|
||||||
|
........................................................................ [ 49%]
|
||||||
|
........................................................................ [ 50%]
|
||||||
|
........................................................................ [ 52%]
|
||||||
|
........................................................................ [ 54%]
|
||||||
|
........................................................................ [ 55%]
|
||||||
|
........................................................................ [ 57%]
|
||||||
|
........................................................................ [ 59%]
|
||||||
|
........................................................................ [ 60%]
|
||||||
|
........................................................................ [ 62%]
|
||||||
|
........................................................................ [ 64%]
|
||||||
|
........................................................................ [ 66%]
|
||||||
|
........................................................................ [ 67%]
|
||||||
|
........................................................................ [ 69%]
|
||||||
|
........................................................................ [ 71%]
|
||||||
|
........................................................................ [ 72%]
|
||||||
|
........................................................................ [ 74%]
|
||||||
|
........................................................................ [ 76%]
|
||||||
|
........................................................................ [ 77%]
|
||||||
|
........................................................................ [ 79%]
|
||||||
|
........................................................................ [ 81%]
|
||||||
|
........................................................................ [ 82%]
|
||||||
|
........................................................................ [ 84%]
|
||||||
|
........................................................................ [ 86%]
|
||||||
|
........................................................................ [ 88%]
|
||||||
|
........................................................................ [ 89%]
|
||||||
|
........................................................................ [ 91%]
|
||||||
|
........................................................................ [ 93%]
|
||||||
|
........................................................................ [ 94%]
|
||||||
|
........................................................................ [ 96%]
|
||||||
|
........................................................................ [ 98%]
|
||||||
|
........................................................................ [ 99%]
|
||||||
|
..... [100%]
|
||||||
|
=============================== warnings summary ===============================
|
||||||
|
.venv/lib/python3.13/site-packages/fastapi/testclient.py:1
|
||||||
|
/var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
|
||||||
|
from starlette.testclient import TestClient as TestClient # noqa
|
||||||
|
|
||||||
|
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
|
||||||
|
================================ tests coverage ================================
|
||||||
|
_______________ coverage: platform linux, python 3.13.13-final-0 _______________
|
||||||
|
|
||||||
|
Name Stmts Miss Cover
|
||||||
|
--------------------------------------------------
|
||||||
|
app/__init__.py 1 0 100%
|
||||||
|
app/api/__init__.py 0 0 100%
|
||||||
|
app/api/admin_settings.py 114 0 100%
|
||||||
|
app/api/auth.py 52 0 100%
|
||||||
|
app/api/chat.py 280 1 99%
|
||||||
|
app/api/chat_images.py 52 0 100%
|
||||||
|
app/api/chats.py 111 0 100%
|
||||||
|
app/api/config.py 13 0 100%
|
||||||
|
app/api/doc_drafts.py 99 0 100%
|
||||||
|
app/api/docs.py 234 1 99%
|
||||||
|
app/api/git_sources.py 259 0 100%
|
||||||
|
app/api/health.py 10 0 100%
|
||||||
|
app/api/steering.py 42 0 100%
|
||||||
|
app/api/suggestions.py 33 0 100%
|
||||||
|
app/api/sync.py 204 0 100%
|
||||||
|
app/api/tokens.py 40 0 100%
|
||||||
|
app/api/ui_settings.py 55 0 100%
|
||||||
|
app/config.py 289 0 100%
|
||||||
|
app/core/__init__.py 0 0 100%
|
||||||
|
app/core/auth.py 45 0 100%
|
||||||
|
app/core/caching.py 124 0 100%
|
||||||
|
app/core/compression.py 59 0 100%
|
||||||
|
app/core/debugging.py 29 2 93%
|
||||||
|
app/core/docs_push.py 43 0 100%
|
||||||
|
app/core/errors.py 5 0 100%
|
||||||
|
app/core/http_cache.py 22 0 100%
|
||||||
|
app/core/logging.py 13 0 100%
|
||||||
|
app/core/rate_limit.py 44 0 100%
|
||||||
|
app/core/response_budget.py 63 0 100%
|
||||||
|
app/core/security_headers.py 20 0 100%
|
||||||
|
app/core/settings_catalog.py 18 0 100%
|
||||||
|
app/core/settings_migrate.py 32 0 100%
|
||||||
|
app/core/settings_overlay.py 111 0 100%
|
||||||
|
app/core/theming.py 40 0 100%
|
||||||
|
app/core/tokens.py 44 0 100%
|
||||||
|
app/db.py 22 0 100%
|
||||||
|
app/main.py 105 0 100%
|
||||||
|
app/models.py 141 0 100%
|
||||||
|
app/rag/__init__.py 0 0 100%
|
||||||
|
app/rag/agent.py 454 1 99%
|
||||||
|
app/rag/archive_upload.py 134 0 100%
|
||||||
|
app/rag/chunker.py 220 4 98%
|
||||||
|
app/rag/doc_dates.py 18 0 100%
|
||||||
|
app/rag/folder_summaries.py 134 0 100%
|
||||||
|
app/rag/git_sources.py 77 0 100%
|
||||||
|
app/rag/importer.py 855 4 99%
|
||||||
|
app/rag/llm.py 253 1 99%
|
||||||
|
app/rag/md_images.py 63 0 100%
|
||||||
|
app/rag/overview.py 78 0 100%
|
||||||
|
app/rag/pdf.py 192 0 100%
|
||||||
|
app/rag/prompts.py 156 0 100%
|
||||||
|
app/rag/retriever.py 252 3 99%
|
||||||
|
app/rag/scaffolding.py 55 0 100%
|
||||||
|
app/rag/source_removal.py 41 0 100%
|
||||||
|
app/rag/sources_meta.py 16 0 100%
|
||||||
|
app/rag/suggestions.py 27 0 100%
|
||||||
|
app/rag/summarizer.py 58 0 100%
|
||||||
|
app/rag/svg_text.py 111 0 100%
|
||||||
|
app/schemas.py 533 0 100%
|
||||||
|
--------------------------------------------------
|
||||||
|
TOTAL 6595 17 99%
|
||||||
|
coverage gate: app/ 99% (>90%) OK
|
||||||
|
All checks passed!
|
||||||
|
0 errors, 0 warnings, 0 informations
|
||||||
|
WARNING: there is a new pyright version available (v1.1.411 -> v1.1.414).
|
||||||
|
Please install the new version or set PYRIGHT_PYTHON_FORCE_VERSION to `latest`
|
||||||
|
|
||||||
|
validation OK
|
||||||
+1
@@ -0,0 +1 @@
|
|||||||
|
(no final assistant message — see the .err log)
|
||||||
+1
@@ -0,0 +1 @@
|
|||||||
|
error: assistant error: 400: {"message":"litellm.BadRequestError: OpenAIException - prompt exceeds the 131072-token context; increase --context or shorten the convers…
|
||||||
+139
@@ -0,0 +1,139 @@
|
|||||||
|
........................................................................ [ 1%]
|
||||||
|
........................................................................ [ 3%]
|
||||||
|
........................................................................ [ 5%]
|
||||||
|
........................................................................ [ 6%]
|
||||||
|
........................................................................ [ 8%]
|
||||||
|
........................................................................ [ 10%]
|
||||||
|
........................................................................ [ 11%]
|
||||||
|
........................................................................ [ 13%]
|
||||||
|
........................................................................ [ 15%]
|
||||||
|
........................................................................ [ 16%]
|
||||||
|
........................................................................ [ 18%]
|
||||||
|
........................................................................ [ 20%]
|
||||||
|
........................................................................ [ 22%]
|
||||||
|
........................................................................ [ 23%]
|
||||||
|
........................................................................ [ 25%]
|
||||||
|
........................................................................ [ 27%]
|
||||||
|
........................................................................ [ 28%]
|
||||||
|
........................................................................ [ 30%]
|
||||||
|
........................................................................ [ 32%]
|
||||||
|
........................................................................ [ 33%]
|
||||||
|
........................................................................ [ 35%]
|
||||||
|
........................................................................ [ 37%]
|
||||||
|
........................................................................ [ 38%]
|
||||||
|
........................................................................ [ 40%]
|
||||||
|
........................................................................ [ 42%]
|
||||||
|
........................................................................ [ 44%]
|
||||||
|
........................................................................ [ 45%]
|
||||||
|
........................................................................ [ 47%]
|
||||||
|
........................................................................ [ 49%]
|
||||||
|
........................................................................ [ 50%]
|
||||||
|
........................................................................ [ 52%]
|
||||||
|
........................................................................ [ 54%]
|
||||||
|
........................................................................ [ 55%]
|
||||||
|
........................................................................ [ 57%]
|
||||||
|
........................................................................ [ 59%]
|
||||||
|
........................................................................ [ 60%]
|
||||||
|
........................................................................ [ 62%]
|
||||||
|
........................................................................ [ 64%]
|
||||||
|
........................................................................ [ 66%]
|
||||||
|
........................................................................ [ 67%]
|
||||||
|
........................................................................ [ 69%]
|
||||||
|
........................................................................ [ 71%]
|
||||||
|
........................................................................ [ 72%]
|
||||||
|
........................................................................ [ 74%]
|
||||||
|
........................................................................ [ 76%]
|
||||||
|
........................................................................ [ 77%]
|
||||||
|
........................................................................ [ 79%]
|
||||||
|
........................................................................ [ 81%]
|
||||||
|
........................................................................ [ 82%]
|
||||||
|
........................................................................ [ 84%]
|
||||||
|
........................................................................ [ 86%]
|
||||||
|
........................................................................ [ 88%]
|
||||||
|
........................................................................ [ 89%]
|
||||||
|
........................................................................ [ 91%]
|
||||||
|
........................................................................ [ 93%]
|
||||||
|
........................................................................ [ 94%]
|
||||||
|
........................................................................ [ 96%]
|
||||||
|
........................................................................ [ 98%]
|
||||||
|
........................................................................ [ 99%]
|
||||||
|
..... [100%]
|
||||||
|
=============================== warnings summary ===============================
|
||||||
|
.venv/lib/python3.13/site-packages/fastapi/testclient.py:1
|
||||||
|
/var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
|
||||||
|
from starlette.testclient import TestClient as TestClient # noqa
|
||||||
|
|
||||||
|
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
|
||||||
|
================================ tests coverage ================================
|
||||||
|
_______________ coverage: platform linux, python 3.13.13-final-0 _______________
|
||||||
|
|
||||||
|
Name Stmts Miss Cover
|
||||||
|
--------------------------------------------------
|
||||||
|
app/__init__.py 1 0 100%
|
||||||
|
app/api/__init__.py 0 0 100%
|
||||||
|
app/api/admin_settings.py 114 0 100%
|
||||||
|
app/api/auth.py 52 0 100%
|
||||||
|
app/api/chat.py 280 1 99%
|
||||||
|
app/api/chat_images.py 52 0 100%
|
||||||
|
app/api/chats.py 111 0 100%
|
||||||
|
app/api/config.py 13 0 100%
|
||||||
|
app/api/doc_drafts.py 99 0 100%
|
||||||
|
app/api/docs.py 234 1 99%
|
||||||
|
app/api/git_sources.py 259 0 100%
|
||||||
|
app/api/health.py 10 0 100%
|
||||||
|
app/api/steering.py 42 0 100%
|
||||||
|
app/api/suggestions.py 33 0 100%
|
||||||
|
app/api/sync.py 204 0 100%
|
||||||
|
app/api/tokens.py 40 0 100%
|
||||||
|
app/api/ui_settings.py 55 0 100%
|
||||||
|
app/config.py 289 0 100%
|
||||||
|
app/core/__init__.py 0 0 100%
|
||||||
|
app/core/auth.py 45 0 100%
|
||||||
|
app/core/caching.py 124 0 100%
|
||||||
|
app/core/compression.py 59 0 100%
|
||||||
|
app/core/debugging.py 29 2 93%
|
||||||
|
app/core/docs_push.py 43 0 100%
|
||||||
|
app/core/errors.py 5 0 100%
|
||||||
|
app/core/http_cache.py 22 0 100%
|
||||||
|
app/core/logging.py 13 0 100%
|
||||||
|
app/core/rate_limit.py 44 0 100%
|
||||||
|
app/core/response_budget.py 63 0 100%
|
||||||
|
app/core/security_headers.py 20 0 100%
|
||||||
|
app/core/settings_catalog.py 18 0 100%
|
||||||
|
app/core/settings_migrate.py 32 0 100%
|
||||||
|
app/core/settings_overlay.py 111 0 100%
|
||||||
|
app/core/theming.py 40 0 100%
|
||||||
|
app/core/tokens.py 44 0 100%
|
||||||
|
app/db.py 22 0 100%
|
||||||
|
app/main.py 105 0 100%
|
||||||
|
app/models.py 141 0 100%
|
||||||
|
app/rag/__init__.py 0 0 100%
|
||||||
|
app/rag/agent.py 454 1 99%
|
||||||
|
app/rag/archive_upload.py 134 0 100%
|
||||||
|
app/rag/chunker.py 220 4 98%
|
||||||
|
app/rag/doc_dates.py 18 0 100%
|
||||||
|
app/rag/folder_summaries.py 134 0 100%
|
||||||
|
app/rag/git_sources.py 77 0 100%
|
||||||
|
app/rag/importer.py 855 4 99%
|
||||||
|
app/rag/llm.py 253 1 99%
|
||||||
|
app/rag/md_images.py 63 0 100%
|
||||||
|
app/rag/overview.py 78 0 100%
|
||||||
|
app/rag/pdf.py 192 0 100%
|
||||||
|
app/rag/prompts.py 156 0 100%
|
||||||
|
app/rag/retriever.py 252 3 99%
|
||||||
|
app/rag/scaffolding.py 55 0 100%
|
||||||
|
app/rag/source_removal.py 41 0 100%
|
||||||
|
app/rag/sources_meta.py 16 0 100%
|
||||||
|
app/rag/suggestions.py 27 0 100%
|
||||||
|
app/rag/summarizer.py 58 0 100%
|
||||||
|
app/rag/svg_text.py 111 0 100%
|
||||||
|
app/schemas.py 533 0 100%
|
||||||
|
--------------------------------------------------
|
||||||
|
TOTAL 6595 17 99%
|
||||||
|
coverage gate: app/ 99% (>90%) OK
|
||||||
|
All checks passed!
|
||||||
|
0 errors, 0 warnings, 0 informations
|
||||||
|
WARNING: there is a new pyright version available (v1.1.411 -> v1.1.414).
|
||||||
|
Please install the new version or set PYRIGHT_PYTHON_FORCE_VERSION to `latest`
|
||||||
|
|
||||||
|
validation OK
|
||||||
+13
@@ -0,0 +1,13 @@
|
|||||||
|
**Implemented (task 03 — SVG bytes route + CSP)**
|
||||||
|
- `app/rag/summarizer.py`: `IMAGE_MIMES[".svg"] = "image/svg+xml"` + doc-comment (the one *vector* entry, bytes-route only — the vector import branch never reaches `image_data_url`).
|
||||||
|
- `app/api/docs.py`: new `SVG_MEDIA_TYPE` (derived from `IMAGE_MIMES[".svg"]`, so guard ≡ Content-Type) and `SVG_SANDBOX_CSP = "default-src 'none'; sandbox"`; `get_document_image` adds that one header **only** when the resolved media type is SVG. Lookup, public id-as-credential posture, four 404 shapes and `Cache-Control: private, max-age=3600` untouched; raster header set byte-identical. Route + module docstrings give the rationale (navigable URL → stored-XSS seam, `sandbox` = opaque origin, middleware preserves an inner CSP, global `nosniff` still applies, no `Content-Disposition`).
|
||||||
|
- New `tests/integration/test_svg_document_image.py` (6 tests: SVG mime+sandbox CSP+exact bytes incl. an unsanitized `<script>` sentinel; pinned header key-set, no `content-disposition`; guard keyed on the map not the client (`.SVG`, unknown-ext fallback → global CSP); PNG response unchanged (`CSP` constant, no `sandbox`); anonymous/public posture; all 404 shapes incl. non-uuid→404 and deleted copy).
|
||||||
|
- `tests/unit/test_image_documents.py`: `IMAGE_MIMES` identity pin extended to `.svg` (six raster entries unchanged); dropped the now-unused `_VECTOR_IMAGE_EXTENSIONS` import (still pinned in `test_chat_image_questions.py`).
|
||||||
|
|
||||||
|
**Gates** — `uv run pytest --cov=app --cov-report=term-missing`: **4259 passed**, TOTAL **99%** (`app/api/docs.py` 99%, `app/rag/summarizer.py` 100%); `uv run pytest tests/integration/test_svg_document_image.py tests/integration/test_docs_api.py -v` → **65 passed**; `uv run ruff check .` → All checks passed; `uv run pyright` → 0 errors. Live check against a real uvicorn: SVG → `content-type: image/svg+xml` + `content-security-policy: default-src 'none'; sandbox`; PNG → `image/png` with no sandbox (seeded rows/copies removed afterwards).
|
||||||
|
|
||||||
|
**Docs** — README: SVG paragraph in *Image documents* gained the "served sandboxed" sentence; the middleware-stack area (where A20 headers are described) gained a **Security headers (A20)** note covering the preserved inner CSP. AGENTS.md: no impact (it documents no header behaviour).
|
||||||
|
|
||||||
|
**Decisions/deviations** — (1) Constants instead of inline literals (values verbatim per the task) so the guard can't drift from the map. (2) The task's "raster → NO `content-security-policy` key" is unachievable: `SecurityHeadersMiddleware` adds the global A20 CSP to every response, so the raster test pins `CSP == app-wide CSP constant`, no `sandbox`, and an identical header key set — same intent (nothing route-specific added). No live-infra changes.
|
||||||
|
|
||||||
|
**Next pending task:** `181_svg_document_preview/04_svg_preview_render.md`.
|
||||||
+1
@@ -0,0 +1 @@
|
|||||||
|
ok
|
||||||
+139
@@ -0,0 +1,139 @@
|
|||||||
|
........................................................................ [ 1%]
|
||||||
|
........................................................................ [ 3%]
|
||||||
|
........................................................................ [ 5%]
|
||||||
|
........................................................................ [ 6%]
|
||||||
|
........................................................................ [ 8%]
|
||||||
|
........................................................................ [ 10%]
|
||||||
|
........................................................................ [ 11%]
|
||||||
|
........................................................................ [ 13%]
|
||||||
|
........................................................................ [ 15%]
|
||||||
|
........................................................................ [ 16%]
|
||||||
|
........................................................................ [ 18%]
|
||||||
|
........................................................................ [ 20%]
|
||||||
|
........................................................................ [ 21%]
|
||||||
|
........................................................................ [ 23%]
|
||||||
|
........................................................................ [ 25%]
|
||||||
|
........................................................................ [ 27%]
|
||||||
|
........................................................................ [ 28%]
|
||||||
|
........................................................................ [ 30%]
|
||||||
|
........................................................................ [ 32%]
|
||||||
|
........................................................................ [ 33%]
|
||||||
|
........................................................................ [ 35%]
|
||||||
|
........................................................................ [ 37%]
|
||||||
|
........................................................................ [ 38%]
|
||||||
|
........................................................................ [ 40%]
|
||||||
|
........................................................................ [ 42%]
|
||||||
|
........................................................................ [ 43%]
|
||||||
|
........................................................................ [ 45%]
|
||||||
|
........................................................................ [ 47%]
|
||||||
|
........................................................................ [ 49%]
|
||||||
|
........................................................................ [ 50%]
|
||||||
|
........................................................................ [ 52%]
|
||||||
|
........................................................................ [ 54%]
|
||||||
|
........................................................................ [ 55%]
|
||||||
|
........................................................................ [ 57%]
|
||||||
|
........................................................................ [ 59%]
|
||||||
|
........................................................................ [ 60%]
|
||||||
|
........................................................................ [ 62%]
|
||||||
|
........................................................................ [ 64%]
|
||||||
|
........................................................................ [ 65%]
|
||||||
|
........................................................................ [ 67%]
|
||||||
|
........................................................................ [ 69%]
|
||||||
|
........................................................................ [ 71%]
|
||||||
|
........................................................................ [ 72%]
|
||||||
|
........................................................................ [ 74%]
|
||||||
|
........................................................................ [ 76%]
|
||||||
|
........................................................................ [ 77%]
|
||||||
|
........................................................................ [ 79%]
|
||||||
|
........................................................................ [ 81%]
|
||||||
|
........................................................................ [ 82%]
|
||||||
|
........................................................................ [ 84%]
|
||||||
|
........................................................................ [ 86%]
|
||||||
|
........................................................................ [ 87%]
|
||||||
|
........................................................................ [ 89%]
|
||||||
|
........................................................................ [ 91%]
|
||||||
|
........................................................................ [ 92%]
|
||||||
|
........................................................................ [ 94%]
|
||||||
|
........................................................................ [ 96%]
|
||||||
|
........................................................................ [ 98%]
|
||||||
|
........................................................................ [ 99%]
|
||||||
|
........... [100%]
|
||||||
|
=============================== warnings summary ===============================
|
||||||
|
.venv/lib/python3.13/site-packages/fastapi/testclient.py:1
|
||||||
|
/var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
|
||||||
|
from starlette.testclient import TestClient as TestClient # noqa
|
||||||
|
|
||||||
|
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
|
||||||
|
================================ tests coverage ================================
|
||||||
|
_______________ coverage: platform linux, python 3.13.13-final-0 _______________
|
||||||
|
|
||||||
|
Name Stmts Miss Cover
|
||||||
|
--------------------------------------------------
|
||||||
|
app/__init__.py 1 0 100%
|
||||||
|
app/api/__init__.py 0 0 100%
|
||||||
|
app/api/admin_settings.py 114 0 100%
|
||||||
|
app/api/auth.py 52 0 100%
|
||||||
|
app/api/chat.py 280 1 99%
|
||||||
|
app/api/chat_images.py 52 0 100%
|
||||||
|
app/api/chats.py 111 0 100%
|
||||||
|
app/api/config.py 13 0 100%
|
||||||
|
app/api/doc_drafts.py 99 0 100%
|
||||||
|
app/api/docs.py 239 1 99%
|
||||||
|
app/api/git_sources.py 259 0 100%
|
||||||
|
app/api/health.py 10 0 100%
|
||||||
|
app/api/steering.py 42 0 100%
|
||||||
|
app/api/suggestions.py 33 0 100%
|
||||||
|
app/api/sync.py 204 0 100%
|
||||||
|
app/api/tokens.py 40 0 100%
|
||||||
|
app/api/ui_settings.py 55 0 100%
|
||||||
|
app/config.py 289 0 100%
|
||||||
|
app/core/__init__.py 0 0 100%
|
||||||
|
app/core/auth.py 45 0 100%
|
||||||
|
app/core/caching.py 124 0 100%
|
||||||
|
app/core/compression.py 59 0 100%
|
||||||
|
app/core/debugging.py 29 2 93%
|
||||||
|
app/core/docs_push.py 43 0 100%
|
||||||
|
app/core/errors.py 5 0 100%
|
||||||
|
app/core/http_cache.py 22 0 100%
|
||||||
|
app/core/logging.py 13 0 100%
|
||||||
|
app/core/rate_limit.py 44 0 100%
|
||||||
|
app/core/response_budget.py 63 0 100%
|
||||||
|
app/core/security_headers.py 20 0 100%
|
||||||
|
app/core/settings_catalog.py 18 0 100%
|
||||||
|
app/core/settings_migrate.py 32 0 100%
|
||||||
|
app/core/settings_overlay.py 111 0 100%
|
||||||
|
app/core/theming.py 40 0 100%
|
||||||
|
app/core/tokens.py 44 0 100%
|
||||||
|
app/db.py 22 0 100%
|
||||||
|
app/main.py 105 0 100%
|
||||||
|
app/models.py 141 0 100%
|
||||||
|
app/rag/__init__.py 0 0 100%
|
||||||
|
app/rag/agent.py 454 1 99%
|
||||||
|
app/rag/archive_upload.py 134 0 100%
|
||||||
|
app/rag/chunker.py 220 4 98%
|
||||||
|
app/rag/doc_dates.py 18 0 100%
|
||||||
|
app/rag/folder_summaries.py 134 0 100%
|
||||||
|
app/rag/git_sources.py 77 0 100%
|
||||||
|
app/rag/importer.py 855 4 99%
|
||||||
|
app/rag/llm.py 253 1 99%
|
||||||
|
app/rag/md_images.py 63 0 100%
|
||||||
|
app/rag/overview.py 78 0 100%
|
||||||
|
app/rag/pdf.py 192 0 100%
|
||||||
|
app/rag/prompts.py 156 0 100%
|
||||||
|
app/rag/retriever.py 252 3 99%
|
||||||
|
app/rag/scaffolding.py 55 0 100%
|
||||||
|
app/rag/source_removal.py 41 0 100%
|
||||||
|
app/rag/sources_meta.py 16 0 100%
|
||||||
|
app/rag/suggestions.py 27 0 100%
|
||||||
|
app/rag/summarizer.py 58 0 100%
|
||||||
|
app/rag/svg_text.py 111 0 100%
|
||||||
|
app/schemas.py 533 0 100%
|
||||||
|
--------------------------------------------------
|
||||||
|
TOTAL 6600 17 99%
|
||||||
|
coverage gate: app/ 99% (>90%) OK
|
||||||
|
All checks passed!
|
||||||
|
0 errors, 0 warnings, 0 informations
|
||||||
|
WARNING: there is a new pyright version available (v1.1.411 -> v1.1.414).
|
||||||
|
Please install the new version or set PYRIGHT_PYTHON_FORCE_VERSION to `latest`
|
||||||
|
|
||||||
|
validation OK
|
||||||
+13
@@ -0,0 +1,13 @@
|
|||||||
|
All gates green. Final report:
|
||||||
|
|
||||||
|
**Implemented (task 04, phase 181):**
|
||||||
|
- `frontend/assets/document.js::docImageBlock` — adds the `doc-image-img--vector` modifier (base class kept) when `doc.format === "svg"`; full rationale comment (no-intrinsic-size → 300×150 default box, `doc.format` is the only type carrier per D7, LOCKED A3 never-inline, legacy-TEXT XML honesty).
|
||||||
|
- `frontend/assets/styles.css` — new `.doc-image-img--vector { width: 100%; height: auto; object-fit: contain; }` beside `.doc-image-img` with house comment; palette-neutral, base rule untouched.
|
||||||
|
- `frontend/assets/app.js::appendSourceImageFigure` — deliberate-non-change comment (96px cap scales size-less SVGs; modal is the preview; no `SourceRef.format` field). **Finding:** no code gap — figure keys off `image_url` only.
|
||||||
|
- `frontend/assets/sources.js` — verified-non-change comment on the `d.is_image` thumbnail branch. **Finding:** no gap found without needing an E2E: `.kb-doc-thumb-img` pins the img to the fixed 48px box (`width/height: 100%` + `object-fit: cover`), and the KB-tree row schema carries `image_url`, so an SVG thumb fills the box like a raster.
|
||||||
|
- `tests/unit/test_document_viewer.py` — docstring + 4 new tests: format-gate source pins, node behavioral run (svg → compound class, `src`=`image_url`, alt=summary/title-fallback incl. blank-summary; png → base class byte-identical), CSS pins (decls, zero colour literals, `object-fit: contain`, base rule intact), and D7 non-change pins for figure/thumbnail.
|
||||||
|
- Docs: README SVG section gains the viewer-sizing sentence + the stale "left to phase 181" forward reference in the phase-180 section updated; AGENTS.md §5 gains the phase-181 layout clause.
|
||||||
|
|
||||||
|
**Gates:** `uv run pytest` → 4263 passed; `uv run pytest --cov=app` → TOTAL 99% (>90%); `uv run ruff check .` → clean; `uv run pyright` → 0 errors; raster suites unedited & green: `tests/e2e/test_image_documents.py` (4 passed), `test_prose_word_wrap.py`+`test_pdf_support.py`+`test_pictorial_pdf.py` (14 passed), `test_chat_answer_clusters.py` (4 passed) — all `--no-cov`, in isolation. Phase E2E file itself is task 05's deliverable.
|
||||||
|
|
||||||
|
**Next pending task:** `181_svg_document_preview/05_unit_integration_e2e.md`.
|
||||||
+1
@@ -0,0 +1 @@
|
|||||||
|
ok
|
||||||
+139
@@ -0,0 +1,139 @@
|
|||||||
|
........................................................................ [ 1%]
|
||||||
|
........................................................................ [ 3%]
|
||||||
|
........................................................................ [ 5%]
|
||||||
|
........................................................................ [ 6%]
|
||||||
|
........................................................................ [ 8%]
|
||||||
|
........................................................................ [ 10%]
|
||||||
|
........................................................................ [ 11%]
|
||||||
|
........................................................................ [ 13%]
|
||||||
|
........................................................................ [ 15%]
|
||||||
|
........................................................................ [ 16%]
|
||||||
|
........................................................................ [ 18%]
|
||||||
|
........................................................................ [ 20%]
|
||||||
|
........................................................................ [ 21%]
|
||||||
|
........................................................................ [ 23%]
|
||||||
|
........................................................................ [ 25%]
|
||||||
|
........................................................................ [ 27%]
|
||||||
|
........................................................................ [ 28%]
|
||||||
|
........................................................................ [ 30%]
|
||||||
|
........................................................................ [ 32%]
|
||||||
|
........................................................................ [ 33%]
|
||||||
|
........................................................................ [ 35%]
|
||||||
|
........................................................................ [ 37%]
|
||||||
|
........................................................................ [ 38%]
|
||||||
|
........................................................................ [ 40%]
|
||||||
|
........................................................................ [ 42%]
|
||||||
|
........................................................................ [ 43%]
|
||||||
|
........................................................................ [ 45%]
|
||||||
|
........................................................................ [ 47%]
|
||||||
|
........................................................................ [ 48%]
|
||||||
|
........................................................................ [ 50%]
|
||||||
|
........................................................................ [ 52%]
|
||||||
|
........................................................................ [ 54%]
|
||||||
|
........................................................................ [ 55%]
|
||||||
|
........................................................................ [ 57%]
|
||||||
|
........................................................................ [ 59%]
|
||||||
|
........................................................................ [ 60%]
|
||||||
|
........................................................................ [ 62%]
|
||||||
|
........................................................................ [ 64%]
|
||||||
|
........................................................................ [ 65%]
|
||||||
|
........................................................................ [ 67%]
|
||||||
|
........................................................................ [ 69%]
|
||||||
|
........................................................................ [ 70%]
|
||||||
|
........................................................................ [ 72%]
|
||||||
|
........................................................................ [ 74%]
|
||||||
|
........................................................................ [ 76%]
|
||||||
|
........................................................................ [ 77%]
|
||||||
|
........................................................................ [ 79%]
|
||||||
|
........................................................................ [ 81%]
|
||||||
|
........................................................................ [ 82%]
|
||||||
|
........................................................................ [ 84%]
|
||||||
|
........................................................................ [ 86%]
|
||||||
|
........................................................................ [ 87%]
|
||||||
|
........................................................................ [ 89%]
|
||||||
|
........................................................................ [ 91%]
|
||||||
|
........................................................................ [ 92%]
|
||||||
|
........................................................................ [ 94%]
|
||||||
|
........................................................................ [ 96%]
|
||||||
|
........................................................................ [ 97%]
|
||||||
|
........................................................................ [ 99%]
|
||||||
|
............... [100%]
|
||||||
|
=============================== warnings summary ===============================
|
||||||
|
.venv/lib/python3.13/site-packages/fastapi/testclient.py:1
|
||||||
|
/var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
|
||||||
|
from starlette.testclient import TestClient as TestClient # noqa
|
||||||
|
|
||||||
|
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
|
||||||
|
================================ tests coverage ================================
|
||||||
|
_______________ coverage: platform linux, python 3.13.13-final-0 _______________
|
||||||
|
|
||||||
|
Name Stmts Miss Cover
|
||||||
|
--------------------------------------------------
|
||||||
|
app/__init__.py 1 0 100%
|
||||||
|
app/api/__init__.py 0 0 100%
|
||||||
|
app/api/admin_settings.py 114 0 100%
|
||||||
|
app/api/auth.py 52 0 100%
|
||||||
|
app/api/chat.py 280 1 99%
|
||||||
|
app/api/chat_images.py 52 0 100%
|
||||||
|
app/api/chats.py 111 0 100%
|
||||||
|
app/api/config.py 13 0 100%
|
||||||
|
app/api/doc_drafts.py 99 0 100%
|
||||||
|
app/api/docs.py 239 1 99%
|
||||||
|
app/api/git_sources.py 259 0 100%
|
||||||
|
app/api/health.py 10 0 100%
|
||||||
|
app/api/steering.py 42 0 100%
|
||||||
|
app/api/suggestions.py 33 0 100%
|
||||||
|
app/api/sync.py 204 0 100%
|
||||||
|
app/api/tokens.py 40 0 100%
|
||||||
|
app/api/ui_settings.py 55 0 100%
|
||||||
|
app/config.py 289 0 100%
|
||||||
|
app/core/__init__.py 0 0 100%
|
||||||
|
app/core/auth.py 45 0 100%
|
||||||
|
app/core/caching.py 124 0 100%
|
||||||
|
app/core/compression.py 59 0 100%
|
||||||
|
app/core/debugging.py 29 2 93%
|
||||||
|
app/core/docs_push.py 43 0 100%
|
||||||
|
app/core/errors.py 5 0 100%
|
||||||
|
app/core/http_cache.py 22 0 100%
|
||||||
|
app/core/logging.py 13 0 100%
|
||||||
|
app/core/rate_limit.py 44 0 100%
|
||||||
|
app/core/response_budget.py 63 0 100%
|
||||||
|
app/core/security_headers.py 20 0 100%
|
||||||
|
app/core/settings_catalog.py 18 0 100%
|
||||||
|
app/core/settings_migrate.py 32 0 100%
|
||||||
|
app/core/settings_overlay.py 111 0 100%
|
||||||
|
app/core/theming.py 40 0 100%
|
||||||
|
app/core/tokens.py 44 0 100%
|
||||||
|
app/db.py 22 0 100%
|
||||||
|
app/main.py 105 0 100%
|
||||||
|
app/models.py 141 0 100%
|
||||||
|
app/rag/__init__.py 0 0 100%
|
||||||
|
app/rag/agent.py 454 1 99%
|
||||||
|
app/rag/archive_upload.py 134 0 100%
|
||||||
|
app/rag/chunker.py 220 4 98%
|
||||||
|
app/rag/doc_dates.py 18 0 100%
|
||||||
|
app/rag/folder_summaries.py 134 0 100%
|
||||||
|
app/rag/git_sources.py 77 0 100%
|
||||||
|
app/rag/importer.py 855 4 99%
|
||||||
|
app/rag/llm.py 253 1 99%
|
||||||
|
app/rag/md_images.py 63 0 100%
|
||||||
|
app/rag/overview.py 78 0 100%
|
||||||
|
app/rag/pdf.py 192 0 100%
|
||||||
|
app/rag/prompts.py 156 0 100%
|
||||||
|
app/rag/retriever.py 252 3 99%
|
||||||
|
app/rag/scaffolding.py 55 0 100%
|
||||||
|
app/rag/source_removal.py 41 0 100%
|
||||||
|
app/rag/sources_meta.py 16 0 100%
|
||||||
|
app/rag/suggestions.py 27 0 100%
|
||||||
|
app/rag/summarizer.py 58 0 100%
|
||||||
|
app/rag/svg_text.py 111 0 100%
|
||||||
|
app/schemas.py 533 0 100%
|
||||||
|
--------------------------------------------------
|
||||||
|
TOTAL 6600 17 99%
|
||||||
|
coverage gate: app/ 99% (>90%) OK
|
||||||
|
All checks passed!
|
||||||
|
0 errors, 0 warnings, 0 informations
|
||||||
|
WARNING: there is a new pyright version available (v1.1.411 -> v1.1.414).
|
||||||
|
Please install the new version or set PYRIGHT_PYTHON_FORCE_VERSION to `latest`
|
||||||
|
|
||||||
|
validation OK
|
||||||
+23
@@ -0,0 +1,23 @@
|
|||||||
|
**Phase 181 task 05 complete — all gates green.**
|
||||||
|
|
||||||
|
**Implemented**
|
||||||
|
- `tests/e2e/test_svg_document_preview.py` (new, dedicated module app with `BOR_IMAGES=true` + suite-pinned `BOR_IMAGE_DIR`, phase-138 truncate-then-`import_sources` seeding, `E2E_REAL_LLM` module skip): `test_viewer_renders_svg_not_xml`, `test_chat_figure_and_modal_preview`, `test_blank_svg_is_indexed_and_searchable`, `test_markdown_image_ref_renders` (two-stage seeding), `test_raster_surfaces_unchanged`
|
||||||
|
- `tests/fixtures/svg_docs/` (new dedicated tree): `diagrams/topology.svg` (viewBox-only, `<title>`/`<desc>`/2 `<text>`, 17 shapes, 2.7 KB), `notes/infra.md` (text co-citation), `diagrams/blank.svg` (pure paths, LOCKED A2 subject)
|
||||||
|
- Extended `tests/integration/test_svg_document_image.py`: content-endpoint SVG contract (`is_image:true`, `image_url` = `/api/documents/<uuid>/image`, `format:"svg"`, `content` = extracted text, serve round trip) + the wire-contract pin that the literal `image_url` passes phase-180's `allowImageUrl` (regex extracted verbatim from `markdown.js`, with rejecting probes)
|
||||||
|
- Tasks 01–04's unit files (`test_svg_text_extraction.py`, `test_svg_image_ingest.py`) verified against the task-05 matrix — all items already pinned; no gaps found
|
||||||
|
|
||||||
|
**Gates (exact results)**
|
||||||
|
- `uv run pytest` → **4265 passed** (re-run after audit DB writes: still 4265 passed)
|
||||||
|
- `uv run pytest --cov=app --cov-report=term-missing` → **TOTAL 99%**, `app/rag/svg_text.py` **100%** (>90% / ≥95% met)
|
||||||
|
- `uv run pytest tests/e2e/test_svg_document_preview.py --no-cov` → **5 passed in isolation**
|
||||||
|
- `uv run ruff check .` → clean; `uv run pyright` → **0 errors**
|
||||||
|
- `uv run python -m scripts.api_payload_audit` (live seeded app) → rc 0, no LOOK rows; SVG route pinned: `:: api method=GET path=/api/documents/<id>/image status=200 bytes=2763 enc=gzip` — far under the 256 KB budget
|
||||||
|
|
||||||
|
**Docs**: no doc impact from this task (README/AGENTS/.env.example were updated by tasks 02/03 and re-verified accurate).
|
||||||
|
|
||||||
|
**Decisions/deviations**
|
||||||
|
- Chat-figure assertion scoped by the figure's `source/path` aria-label: the seed legitimately auto-cites blank.svg too (phase-124 auto-cite), so count-1 is asserted on the topology figure specifically.
|
||||||
|
- Fixture XML comment must avoid `--` (illegal in comments) — caught and fixed; keeps the fixture honestly parseable.
|
||||||
|
- No commit/move performed: per the harness's binding rule the pipeline makes the one atomic commit (`feat(kb): index SVGs as image documents and preview them instead of raw XML`, `app/ frontend/ tests/ .env.example .agents/`) and gates the phase-dir move; everything is left staged-ready in the working tree.
|
||||||
|
|
||||||
|
**Next pending task**: none in phase 181 (05 was the last); next todo phase is `182_verbatim_show_text`.
|
||||||
+1
@@ -0,0 +1 @@
|
|||||||
|
ok
|
||||||
+139
@@ -0,0 +1,139 @@
|
|||||||
|
........................................................................ [ 1%]
|
||||||
|
........................................................................ [ 3%]
|
||||||
|
........................................................................ [ 5%]
|
||||||
|
........................................................................ [ 6%]
|
||||||
|
........................................................................ [ 8%]
|
||||||
|
........................................................................ [ 10%]
|
||||||
|
........................................................................ [ 11%]
|
||||||
|
........................................................................ [ 13%]
|
||||||
|
........................................................................ [ 15%]
|
||||||
|
........................................................................ [ 16%]
|
||||||
|
........................................................................ [ 18%]
|
||||||
|
........................................................................ [ 20%]
|
||||||
|
........................................................................ [ 21%]
|
||||||
|
........................................................................ [ 23%]
|
||||||
|
........................................................................ [ 25%]
|
||||||
|
........................................................................ [ 27%]
|
||||||
|
........................................................................ [ 28%]
|
||||||
|
........................................................................ [ 30%]
|
||||||
|
........................................................................ [ 32%]
|
||||||
|
........................................................................ [ 33%]
|
||||||
|
........................................................................ [ 35%]
|
||||||
|
........................................................................ [ 37%]
|
||||||
|
........................................................................ [ 38%]
|
||||||
|
........................................................................ [ 40%]
|
||||||
|
........................................................................ [ 42%]
|
||||||
|
........................................................................ [ 43%]
|
||||||
|
........................................................................ [ 45%]
|
||||||
|
........................................................................ [ 47%]
|
||||||
|
........................................................................ [ 48%]
|
||||||
|
........................................................................ [ 50%]
|
||||||
|
........................................................................ [ 52%]
|
||||||
|
........................................................................ [ 54%]
|
||||||
|
........................................................................ [ 55%]
|
||||||
|
........................................................................ [ 57%]
|
||||||
|
........................................................................ [ 59%]
|
||||||
|
........................................................................ [ 60%]
|
||||||
|
........................................................................ [ 62%]
|
||||||
|
........................................................................ [ 64%]
|
||||||
|
........................................................................ [ 65%]
|
||||||
|
........................................................................ [ 67%]
|
||||||
|
........................................................................ [ 69%]
|
||||||
|
........................................................................ [ 70%]
|
||||||
|
........................................................................ [ 72%]
|
||||||
|
........................................................................ [ 74%]
|
||||||
|
........................................................................ [ 75%]
|
||||||
|
........................................................................ [ 77%]
|
||||||
|
........................................................................ [ 79%]
|
||||||
|
........................................................................ [ 81%]
|
||||||
|
........................................................................ [ 82%]
|
||||||
|
........................................................................ [ 84%]
|
||||||
|
........................................................................ [ 86%]
|
||||||
|
........................................................................ [ 87%]
|
||||||
|
........................................................................ [ 89%]
|
||||||
|
........................................................................ [ 91%]
|
||||||
|
........................................................................ [ 92%]
|
||||||
|
........................................................................ [ 94%]
|
||||||
|
........................................................................ [ 96%]
|
||||||
|
........................................................................ [ 97%]
|
||||||
|
........................................................................ [ 99%]
|
||||||
|
................. [100%]
|
||||||
|
=============================== warnings summary ===============================
|
||||||
|
.venv/lib/python3.13/site-packages/fastapi/testclient.py:1
|
||||||
|
/var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
|
||||||
|
from starlette.testclient import TestClient as TestClient # noqa
|
||||||
|
|
||||||
|
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
|
||||||
|
================================ tests coverage ================================
|
||||||
|
_______________ coverage: platform linux, python 3.13.13-final-0 _______________
|
||||||
|
|
||||||
|
Name Stmts Miss Cover
|
||||||
|
--------------------------------------------------
|
||||||
|
app/__init__.py 1 0 100%
|
||||||
|
app/api/__init__.py 0 0 100%
|
||||||
|
app/api/admin_settings.py 114 0 100%
|
||||||
|
app/api/auth.py 52 0 100%
|
||||||
|
app/api/chat.py 280 1 99%
|
||||||
|
app/api/chat_images.py 52 0 100%
|
||||||
|
app/api/chats.py 111 0 100%
|
||||||
|
app/api/config.py 13 0 100%
|
||||||
|
app/api/doc_drafts.py 99 0 100%
|
||||||
|
app/api/docs.py 239 1 99%
|
||||||
|
app/api/git_sources.py 259 0 100%
|
||||||
|
app/api/health.py 10 0 100%
|
||||||
|
app/api/steering.py 42 0 100%
|
||||||
|
app/api/suggestions.py 33 0 100%
|
||||||
|
app/api/sync.py 204 0 100%
|
||||||
|
app/api/tokens.py 40 0 100%
|
||||||
|
app/api/ui_settings.py 55 0 100%
|
||||||
|
app/config.py 289 0 100%
|
||||||
|
app/core/__init__.py 0 0 100%
|
||||||
|
app/core/auth.py 45 0 100%
|
||||||
|
app/core/caching.py 124 0 100%
|
||||||
|
app/core/compression.py 59 0 100%
|
||||||
|
app/core/debugging.py 29 2 93%
|
||||||
|
app/core/docs_push.py 43 0 100%
|
||||||
|
app/core/errors.py 5 0 100%
|
||||||
|
app/core/http_cache.py 22 0 100%
|
||||||
|
app/core/logging.py 13 0 100%
|
||||||
|
app/core/rate_limit.py 44 0 100%
|
||||||
|
app/core/response_budget.py 63 0 100%
|
||||||
|
app/core/security_headers.py 20 0 100%
|
||||||
|
app/core/settings_catalog.py 18 0 100%
|
||||||
|
app/core/settings_migrate.py 32 0 100%
|
||||||
|
app/core/settings_overlay.py 111 0 100%
|
||||||
|
app/core/theming.py 40 0 100%
|
||||||
|
app/core/tokens.py 44 0 100%
|
||||||
|
app/db.py 22 0 100%
|
||||||
|
app/main.py 105 0 100%
|
||||||
|
app/models.py 141 0 100%
|
||||||
|
app/rag/__init__.py 0 0 100%
|
||||||
|
app/rag/agent.py 454 1 99%
|
||||||
|
app/rag/archive_upload.py 134 0 100%
|
||||||
|
app/rag/chunker.py 220 4 98%
|
||||||
|
app/rag/doc_dates.py 18 0 100%
|
||||||
|
app/rag/folder_summaries.py 134 0 100%
|
||||||
|
app/rag/git_sources.py 77 0 100%
|
||||||
|
app/rag/importer.py 855 4 99%
|
||||||
|
app/rag/llm.py 253 1 99%
|
||||||
|
app/rag/md_images.py 63 0 100%
|
||||||
|
app/rag/overview.py 78 0 100%
|
||||||
|
app/rag/pdf.py 192 0 100%
|
||||||
|
app/rag/prompts.py 156 0 100%
|
||||||
|
app/rag/retriever.py 252 3 99%
|
||||||
|
app/rag/scaffolding.py 55 0 100%
|
||||||
|
app/rag/source_removal.py 41 0 100%
|
||||||
|
app/rag/sources_meta.py 16 0 100%
|
||||||
|
app/rag/suggestions.py 27 0 100%
|
||||||
|
app/rag/summarizer.py 58 0 100%
|
||||||
|
app/rag/svg_text.py 111 0 100%
|
||||||
|
app/schemas.py 533 0 100%
|
||||||
|
--------------------------------------------------
|
||||||
|
TOTAL 6600 17 99%
|
||||||
|
coverage gate: app/ 99% (>90%) OK
|
||||||
|
All checks passed!
|
||||||
|
0 errors, 0 warnings, 0 informations
|
||||||
|
WARNING: there is a new pyright version available (v1.1.411 -> v1.1.414).
|
||||||
|
Please install the new version or set PYRIGHT_PYTHON_FORCE_VERSION to `latest`
|
||||||
|
|
||||||
|
validation OK
|
||||||
+1
@@ -0,0 +1 @@
|
|||||||
|
(no final assistant message — see the .err log)
|
||||||
+1
@@ -0,0 +1 @@
|
|||||||
|
error: assistant error: 400: {"message":"litellm.BadRequestError: OpenAIException - prompt exceeds the 131072-token context; increase --context or shorten the convers…
|
||||||
+139
@@ -0,0 +1,139 @@
|
|||||||
|
........................................................................ [ 1%]
|
||||||
|
........................................................................ [ 3%]
|
||||||
|
........................................................................ [ 5%]
|
||||||
|
........................................................................ [ 6%]
|
||||||
|
........................................................................ [ 8%]
|
||||||
|
........................................................................ [ 10%]
|
||||||
|
........................................................................ [ 11%]
|
||||||
|
........................................................................ [ 13%]
|
||||||
|
........................................................................ [ 15%]
|
||||||
|
........................................................................ [ 16%]
|
||||||
|
........................................................................ [ 18%]
|
||||||
|
........................................................................ [ 20%]
|
||||||
|
........................................................................ [ 21%]
|
||||||
|
........................................................................ [ 23%]
|
||||||
|
........................................................................ [ 25%]
|
||||||
|
........................................................................ [ 27%]
|
||||||
|
........................................................................ [ 28%]
|
||||||
|
........................................................................ [ 30%]
|
||||||
|
........................................................................ [ 32%]
|
||||||
|
........................................................................ [ 33%]
|
||||||
|
........................................................................ [ 35%]
|
||||||
|
........................................................................ [ 37%]
|
||||||
|
........................................................................ [ 38%]
|
||||||
|
........................................................................ [ 40%]
|
||||||
|
........................................................................ [ 42%]
|
||||||
|
........................................................................ [ 43%]
|
||||||
|
........................................................................ [ 45%]
|
||||||
|
........................................................................ [ 47%]
|
||||||
|
........................................................................ [ 48%]
|
||||||
|
........................................................................ [ 50%]
|
||||||
|
........................................................................ [ 52%]
|
||||||
|
........................................................................ [ 54%]
|
||||||
|
........................................................................ [ 55%]
|
||||||
|
........................................................................ [ 57%]
|
||||||
|
........................................................................ [ 59%]
|
||||||
|
........................................................................ [ 60%]
|
||||||
|
........................................................................ [ 62%]
|
||||||
|
........................................................................ [ 64%]
|
||||||
|
........................................................................ [ 65%]
|
||||||
|
........................................................................ [ 67%]
|
||||||
|
........................................................................ [ 69%]
|
||||||
|
........................................................................ [ 70%]
|
||||||
|
........................................................................ [ 72%]
|
||||||
|
........................................................................ [ 74%]
|
||||||
|
........................................................................ [ 76%]
|
||||||
|
........................................................................ [ 77%]
|
||||||
|
........................................................................ [ 79%]
|
||||||
|
........................................................................ [ 81%]
|
||||||
|
........................................................................ [ 82%]
|
||||||
|
........................................................................ [ 84%]
|
||||||
|
........................................................................ [ 86%]
|
||||||
|
........................................................................ [ 87%]
|
||||||
|
........................................................................ [ 89%]
|
||||||
|
........................................................................ [ 91%]
|
||||||
|
........................................................................ [ 92%]
|
||||||
|
........................................................................ [ 94%]
|
||||||
|
........................................................................ [ 96%]
|
||||||
|
........................................................................ [ 97%]
|
||||||
|
........................................................................ [ 99%]
|
||||||
|
............... [100%]
|
||||||
|
=============================== warnings summary ===============================
|
||||||
|
.venv/lib/python3.13/site-packages/fastapi/testclient.py:1
|
||||||
|
/var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
|
||||||
|
from starlette.testclient import TestClient as TestClient # noqa
|
||||||
|
|
||||||
|
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
|
||||||
|
================================ tests coverage ================================
|
||||||
|
_______________ coverage: platform linux, python 3.13.13-final-0 _______________
|
||||||
|
|
||||||
|
Name Stmts Miss Cover
|
||||||
|
--------------------------------------------------
|
||||||
|
app/__init__.py 1 0 100%
|
||||||
|
app/api/__init__.py 0 0 100%
|
||||||
|
app/api/admin_settings.py 114 0 100%
|
||||||
|
app/api/auth.py 52 0 100%
|
||||||
|
app/api/chat.py 280 1 99%
|
||||||
|
app/api/chat_images.py 52 0 100%
|
||||||
|
app/api/chats.py 111 0 100%
|
||||||
|
app/api/config.py 13 0 100%
|
||||||
|
app/api/doc_drafts.py 99 0 100%
|
||||||
|
app/api/docs.py 239 1 99%
|
||||||
|
app/api/git_sources.py 259 0 100%
|
||||||
|
app/api/health.py 10 0 100%
|
||||||
|
app/api/steering.py 42 0 100%
|
||||||
|
app/api/suggestions.py 33 0 100%
|
||||||
|
app/api/sync.py 204 0 100%
|
||||||
|
app/api/tokens.py 40 0 100%
|
||||||
|
app/api/ui_settings.py 55 0 100%
|
||||||
|
app/config.py 289 0 100%
|
||||||
|
app/core/__init__.py 0 0 100%
|
||||||
|
app/core/auth.py 45 0 100%
|
||||||
|
app/core/caching.py 124 0 100%
|
||||||
|
app/core/compression.py 59 0 100%
|
||||||
|
app/core/debugging.py 29 2 93%
|
||||||
|
app/core/docs_push.py 43 0 100%
|
||||||
|
app/core/errors.py 5 0 100%
|
||||||
|
app/core/http_cache.py 22 0 100%
|
||||||
|
app/core/logging.py 13 0 100%
|
||||||
|
app/core/rate_limit.py 44 0 100%
|
||||||
|
app/core/response_budget.py 63 0 100%
|
||||||
|
app/core/security_headers.py 20 0 100%
|
||||||
|
app/core/settings_catalog.py 18 0 100%
|
||||||
|
app/core/settings_migrate.py 32 0 100%
|
||||||
|
app/core/settings_overlay.py 111 0 100%
|
||||||
|
app/core/theming.py 40 0 100%
|
||||||
|
app/core/tokens.py 44 0 100%
|
||||||
|
app/db.py 22 0 100%
|
||||||
|
app/main.py 105 0 100%
|
||||||
|
app/models.py 141 0 100%
|
||||||
|
app/rag/__init__.py 0 0 100%
|
||||||
|
app/rag/agent.py 454 1 99%
|
||||||
|
app/rag/archive_upload.py 134 0 100%
|
||||||
|
app/rag/chunker.py 220 4 98%
|
||||||
|
app/rag/doc_dates.py 18 0 100%
|
||||||
|
app/rag/folder_summaries.py 134 0 100%
|
||||||
|
app/rag/git_sources.py 77 0 100%
|
||||||
|
app/rag/importer.py 855 4 99%
|
||||||
|
app/rag/llm.py 253 1 99%
|
||||||
|
app/rag/md_images.py 63 0 100%
|
||||||
|
app/rag/overview.py 78 0 100%
|
||||||
|
app/rag/pdf.py 192 0 100%
|
||||||
|
app/rag/prompts.py 156 0 100%
|
||||||
|
app/rag/retriever.py 252 3 99%
|
||||||
|
app/rag/scaffolding.py 55 0 100%
|
||||||
|
app/rag/source_removal.py 41 0 100%
|
||||||
|
app/rag/sources_meta.py 16 0 100%
|
||||||
|
app/rag/suggestions.py 27 0 100%
|
||||||
|
app/rag/summarizer.py 58 0 100%
|
||||||
|
app/rag/svg_text.py 111 0 100%
|
||||||
|
app/schemas.py 533 0 100%
|
||||||
|
--------------------------------------------------
|
||||||
|
TOTAL 6600 17 99%
|
||||||
|
coverage gate: app/ 99% (>90%) OK
|
||||||
|
All checks passed!
|
||||||
|
0 errors, 0 warnings, 0 informations
|
||||||
|
WARNING: there is a new pyright version available (v1.1.411 -> v1.1.414).
|
||||||
|
Please install the new version or set PYRIGHT_PYTHON_FORCE_VERSION to `latest`
|
||||||
|
|
||||||
|
validation OK
|
||||||
+1
@@ -0,0 +1 @@
|
|||||||
|
(no final assistant message — see the .err log)
|
||||||
+1
@@ -0,0 +1 @@
|
|||||||
|
error: assistant error: 400: {"message":"litellm.BadRequestError: OpenAIException - prompt exceeds the 131072-token context; increase --context or shorten the convers…
|
||||||
+139
@@ -0,0 +1,139 @@
|
|||||||
|
........................................................................ [ 1%]
|
||||||
|
........................................................................ [ 3%]
|
||||||
|
........................................................................ [ 5%]
|
||||||
|
........................................................................ [ 6%]
|
||||||
|
........................................................................ [ 8%]
|
||||||
|
........................................................................ [ 10%]
|
||||||
|
........................................................................ [ 11%]
|
||||||
|
........................................................................ [ 13%]
|
||||||
|
........................................................................ [ 15%]
|
||||||
|
........................................................................ [ 16%]
|
||||||
|
........................................................................ [ 18%]
|
||||||
|
........................................................................ [ 20%]
|
||||||
|
........................................................................ [ 21%]
|
||||||
|
........................................................................ [ 23%]
|
||||||
|
........................................................................ [ 25%]
|
||||||
|
........................................................................ [ 27%]
|
||||||
|
........................................................................ [ 28%]
|
||||||
|
........................................................................ [ 30%]
|
||||||
|
........................................................................ [ 32%]
|
||||||
|
........................................................................ [ 33%]
|
||||||
|
........................................................................ [ 35%]
|
||||||
|
........................................................................ [ 37%]
|
||||||
|
........................................................................ [ 38%]
|
||||||
|
........................................................................ [ 40%]
|
||||||
|
........................................................................ [ 42%]
|
||||||
|
........................................................................ [ 43%]
|
||||||
|
........................................................................ [ 45%]
|
||||||
|
........................................................................ [ 47%]
|
||||||
|
........................................................................ [ 48%]
|
||||||
|
........................................................................ [ 50%]
|
||||||
|
........................................................................ [ 52%]
|
||||||
|
........................................................................ [ 54%]
|
||||||
|
........................................................................ [ 55%]
|
||||||
|
........................................................................ [ 57%]
|
||||||
|
........................................................................ [ 59%]
|
||||||
|
........................................................................ [ 60%]
|
||||||
|
........................................................................ [ 62%]
|
||||||
|
........................................................................ [ 64%]
|
||||||
|
........................................................................ [ 65%]
|
||||||
|
........................................................................ [ 67%]
|
||||||
|
........................................................................ [ 69%]
|
||||||
|
........................................................................ [ 70%]
|
||||||
|
........................................................................ [ 72%]
|
||||||
|
........................................................................ [ 74%]
|
||||||
|
........................................................................ [ 76%]
|
||||||
|
........................................................................ [ 77%]
|
||||||
|
........................................................................ [ 79%]
|
||||||
|
........................................................................ [ 81%]
|
||||||
|
........................................................................ [ 82%]
|
||||||
|
........................................................................ [ 84%]
|
||||||
|
........................................................................ [ 86%]
|
||||||
|
........................................................................ [ 87%]
|
||||||
|
........................................................................ [ 89%]
|
||||||
|
........................................................................ [ 91%]
|
||||||
|
........................................................................ [ 92%]
|
||||||
|
........................................................................ [ 94%]
|
||||||
|
........................................................................ [ 96%]
|
||||||
|
........................................................................ [ 97%]
|
||||||
|
........................................................................ [ 99%]
|
||||||
|
............... [100%]
|
||||||
|
=============================== warnings summary ===============================
|
||||||
|
.venv/lib/python3.13/site-packages/fastapi/testclient.py:1
|
||||||
|
/var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
|
||||||
|
from starlette.testclient import TestClient as TestClient # noqa
|
||||||
|
|
||||||
|
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
|
||||||
|
================================ tests coverage ================================
|
||||||
|
_______________ coverage: platform linux, python 3.13.13-final-0 _______________
|
||||||
|
|
||||||
|
Name Stmts Miss Cover
|
||||||
|
--------------------------------------------------
|
||||||
|
app/__init__.py 1 0 100%
|
||||||
|
app/api/__init__.py 0 0 100%
|
||||||
|
app/api/admin_settings.py 114 0 100%
|
||||||
|
app/api/auth.py 52 0 100%
|
||||||
|
app/api/chat.py 280 1 99%
|
||||||
|
app/api/chat_images.py 52 0 100%
|
||||||
|
app/api/chats.py 111 0 100%
|
||||||
|
app/api/config.py 13 0 100%
|
||||||
|
app/api/doc_drafts.py 99 0 100%
|
||||||
|
app/api/docs.py 239 1 99%
|
||||||
|
app/api/git_sources.py 259 0 100%
|
||||||
|
app/api/health.py 10 0 100%
|
||||||
|
app/api/steering.py 42 0 100%
|
||||||
|
app/api/suggestions.py 33 0 100%
|
||||||
|
app/api/sync.py 204 0 100%
|
||||||
|
app/api/tokens.py 40 0 100%
|
||||||
|
app/api/ui_settings.py 55 0 100%
|
||||||
|
app/config.py 289 0 100%
|
||||||
|
app/core/__init__.py 0 0 100%
|
||||||
|
app/core/auth.py 45 0 100%
|
||||||
|
app/core/caching.py 124 0 100%
|
||||||
|
app/core/compression.py 59 0 100%
|
||||||
|
app/core/debugging.py 29 2 93%
|
||||||
|
app/core/docs_push.py 43 0 100%
|
||||||
|
app/core/errors.py 5 0 100%
|
||||||
|
app/core/http_cache.py 22 0 100%
|
||||||
|
app/core/logging.py 13 0 100%
|
||||||
|
app/core/rate_limit.py 44 0 100%
|
||||||
|
app/core/response_budget.py 63 0 100%
|
||||||
|
app/core/security_headers.py 20 0 100%
|
||||||
|
app/core/settings_catalog.py 18 0 100%
|
||||||
|
app/core/settings_migrate.py 32 0 100%
|
||||||
|
app/core/settings_overlay.py 111 0 100%
|
||||||
|
app/core/theming.py 40 0 100%
|
||||||
|
app/core/tokens.py 44 0 100%
|
||||||
|
app/db.py 22 0 100%
|
||||||
|
app/main.py 105 0 100%
|
||||||
|
app/models.py 141 0 100%
|
||||||
|
app/rag/__init__.py 0 0 100%
|
||||||
|
app/rag/agent.py 454 1 99%
|
||||||
|
app/rag/archive_upload.py 134 0 100%
|
||||||
|
app/rag/chunker.py 220 4 98%
|
||||||
|
app/rag/doc_dates.py 18 0 100%
|
||||||
|
app/rag/folder_summaries.py 134 0 100%
|
||||||
|
app/rag/git_sources.py 77 0 100%
|
||||||
|
app/rag/importer.py 855 4 99%
|
||||||
|
app/rag/llm.py 253 1 99%
|
||||||
|
app/rag/md_images.py 63 0 100%
|
||||||
|
app/rag/overview.py 78 0 100%
|
||||||
|
app/rag/pdf.py 192 0 100%
|
||||||
|
app/rag/prompts.py 156 0 100%
|
||||||
|
app/rag/retriever.py 252 3 99%
|
||||||
|
app/rag/scaffolding.py 55 0 100%
|
||||||
|
app/rag/source_removal.py 41 0 100%
|
||||||
|
app/rag/sources_meta.py 16 0 100%
|
||||||
|
app/rag/suggestions.py 27 0 100%
|
||||||
|
app/rag/summarizer.py 58 0 100%
|
||||||
|
app/rag/svg_text.py 111 0 100%
|
||||||
|
app/schemas.py 533 0 100%
|
||||||
|
--------------------------------------------------
|
||||||
|
TOTAL 6600 17 99%
|
||||||
|
coverage gate: app/ 99% (>90%) OK
|
||||||
|
All checks passed!
|
||||||
|
0 errors, 0 warnings, 0 informations
|
||||||
|
WARNING: there is a new pyright version available (v1.1.411 -> v1.1.414).
|
||||||
|
Please install the new version or set PYRIGHT_PYTHON_FORCE_VERSION to `latest`
|
||||||
|
|
||||||
|
validation OK
|
||||||
+9
-1
@@ -164,7 +164,15 @@ BOR_IMPORT_EXTENSIONS=md,markdown,txt,yaml,yml,json,py,container,network,volume,
|
|||||||
# ignored by the import walks and a sync never prunes existing image
|
# ignored by the import walks and a sync never prunes existing image
|
||||||
# documents.
|
# documents.
|
||||||
# BOR_IMAGES=0 # 1 = index standalone images
|
# BOR_IMAGES=0 # 1 = index standalone images
|
||||||
# BOR_IMAGE_EXTENSIONS=png,jpg,jpeg,webp,gif,bmp # comma-separated, case-insensitive
|
# BOR_IMAGE_EXTENSIONS=png,jpg,jpeg,webp,gif,bmp,svg # comma-separated, case-insensitive
|
||||||
|
# # svg is the one VECTOR member: an .svg indexes from its own
|
||||||
|
# # <title>/<desc>/<text> (or a structural line when it carries no
|
||||||
|
# # words) and is NEVER sent to the vision model; it is served as
|
||||||
|
# # image/svg+xml so the chat figure, the modal and the document
|
||||||
|
# # viewer show the picture instead of the XML. With BOR_IMAGES=0 an
|
||||||
|
# # .svg stays a plain text document (raw XML in the viewer).
|
||||||
|
# # Question images (the paperclip) stay raster-only: an .svg upload
|
||||||
|
# # is a 422.
|
||||||
# BOR_IMAGE_DIR=~/bor-sources/images # persistent home for the served image bytes
|
# BOR_IMAGE_DIR=~/bor-sources/images # persistent home for the served image bytes
|
||||||
# (uploads are replaced, checkouts re-cloned)
|
# (uploads are replaced, checkouts re-cloned)
|
||||||
|
|
||||||
|
|||||||
@@ -30,7 +30,14 @@
|
|||||||
`javascript:`/`data:` element), a container-capped
|
`javascript:`/`data:` element), a container-capped
|
||||||
`img.md-img` (`max-width: 100%` — no horizontal overflow at 375px)
|
`img.md-img` (`max-width: 100%` — no horizontal overflow at 375px)
|
||||||
and an `h5`/`h6` sized into the existing type scale — never printed
|
and an `h5`/`h6` sized into the existing type scale — never printed
|
||||||
reference text (phase 180); the document viewer's page strip is
|
reference text (phase 180); an SVG document paints as a
|
||||||
|
graphic — the viewer's image block adds
|
||||||
|
`.doc-image-img--vector` (`width: 100%`, `object-fit: contain`)
|
||||||
|
when `format` is `svg` so a `viewBox`-only drawing spans the
|
||||||
|
`.doc-image` card instead of the 300×150 default box, while the
|
||||||
|
chat figure and Sources thumbnail keep their fixed caps and the
|
||||||
|
bytes stay an `<img>` src (never inlined) (phase 181); the
|
||||||
|
document viewer's page strip is
|
||||||
CLASS-scoped —
|
CLASS-scoped —
|
||||||
the primary class shows the strip in place of the text, the
|
the primary class shows the strip in place of the text, the
|
||||||
additive class never hides the text: Text/Pages toggle group
|
additive class never hides the text: Text/Pages toggle group
|
||||||
|
|||||||
@@ -141,8 +141,10 @@ string, so it carries no inline `onerror` handler — the JS fallbacks live
|
|||||||
in `document.js`/`app.js`). `.md-img` carries the app's bordered-image card
|
in `document.js`/`app.js`). `.md-img` carries the app's bordered-image card
|
||||||
treatment and is capped to its container — `max-width: 100%`, `height: auto`
|
treatment and is capped to its container — `max-width: 100%`, `height: auto`
|
||||||
and `object-fit: contain` — so a wide screenshot never produces a horizontal
|
and `object-fit: contain` — so a wide screenshot never produces a horizontal
|
||||||
scrollbar at 375px or at 1440px. Sizing an SVG that declares no `width` or
|
scrollbar at 375px or at 1440px. An SVG document that declares no `width`
|
||||||
`height` is deliberately left to phase 181.
|
or `height` is sized by phase 181's `.doc-image-img--vector` class in the
|
||||||
|
document viewer instead (see "Image documents"); an inline answer image
|
||||||
|
keeps the container cap above.
|
||||||
|
|
||||||
**The heading ladder reaches six levels** (phase 180, TODO.md L5): a
|
**The heading ladder reaches six levels** (phase 180, TODO.md L5): a
|
||||||
`#### x` line in an answer, the thinking block, a saved/shared chat, the
|
`#### x` line in an answer, the thinking block, a saved/shared chat, the
|
||||||
@@ -436,7 +438,9 @@ reflect the new files.
|
|||||||
- Non-markdown files get format-aware chunking (YAML keys, JSON keys,
|
- Non-markdown files get format-aware chunking (YAML keys, JSON keys,
|
||||||
Python classes/defs).
|
Python classes/defs).
|
||||||
- With `BOR_IMAGES=1`, standalone images (`png, jpg, jpeg, webp, gif,
|
- With `BOR_IMAGES=1`, standalone images (`png, jpg, jpeg, webp, gif,
|
||||||
bmp`) are indexed as documents too — see
|
bmp, svg`) are indexed as documents too — `svg` is the one **vector**
|
||||||
|
member, indexed from its own markup text with no vision call and
|
||||||
|
previewed as a graphic (phase 181) — see
|
||||||
[Image support](#image-support).
|
[Image support](#image-support).
|
||||||
- A run that changed the KB also regenerates the **KB overview** and the
|
- A run that changed the KB also regenerates the **KB overview** and the
|
||||||
**folder summaries** (the drill-down catalog's stored per-folder
|
**folder summaries** (the drill-down catalog's stored per-folder
|
||||||
@@ -858,10 +862,46 @@ first-class document:
|
|||||||
row, no copy), logs a warning, and counts in the import summary line
|
row, no copy), logs a warning, and counts in the import summary line
|
||||||
(`images_failed=N`) — the sync continues.
|
(`images_failed=N`) — the sync continues.
|
||||||
|
|
||||||
Accepted formats are `png, jpg, jpeg, webp, gif, bmp`
|
Accepted formats are `png, jpg, jpeg, webp, gif, bmp, svg`
|
||||||
(`BOR_IMAGE_EXTENSIONS`, case-insensitive) — a separate set from
|
(`BOR_IMAGE_EXTENSIONS`, case-insensitive) — a separate set from
|
||||||
`BOR_IMPORT_EXTENSIONS`; images are never added via that variable, the
|
`BOR_IMPORT_EXTENSIONS`; images are never added via that variable, the
|
||||||
toggle is the single knob.
|
toggle is the single knob. With `BOR_IMAGES=0` an `.svg` is therefore a
|
||||||
|
plain **text** document — raw XML in the viewer, no image copy, no
|
||||||
|
thumbnail — and the walks never admit it as an image (no special-casing
|
||||||
|
of the off state).
|
||||||
|
|
||||||
|
**SVG is the one vector member (phase 181).** An `.svg` takes the same
|
||||||
|
image path — its bytes are copied to `BOR_IMAGE_DIR` and served as
|
||||||
|
`image/svg+xml`, so the chat figure, the document modal and the
|
||||||
|
`/document.html` viewer show the *picture* instead of the XML — but it
|
||||||
|
is **never sent to the vision model** (vision endpoints reject
|
||||||
|
`image/svg+xml`). Its searchable text comes out of the file itself: the
|
||||||
|
`<title>`, `<desc>` and `<text>` words, or — for a drawing that carries
|
||||||
|
no words at all — one honest structural line built from the file (its
|
||||||
|
name, its shape census, its `viewBox`). A file the extractor refuses
|
||||||
|
(over 2 MiB, carrying a `<!DOCTYPE`/`<!ENTITY` declaration, or malformed
|
||||||
|
XML) fails soft like any other image: no row, no copy, `images_failed`.
|
||||||
|
A successful one logs one `import: svg indexed source=… path=…
|
||||||
|
structural=… chars=…` line and adds to the log-only `svg_indexed=N` on
|
||||||
|
the import summary line (`added`/`updated` count it like any other image
|
||||||
|
doc). The next sync also **converts** an `.svg` that an earlier run
|
||||||
|
indexed as plain text (its bytes hash is unchanged, so a hash-only fast
|
||||||
|
path would freeze it as XML forever) — existing libraries need no
|
||||||
|
re-import by hand. In the viewer the graphic **spans the image card**: a
|
||||||
|
`viewBox`-only SVG
|
||||||
|
carries no intrinsic size, so `doc.format === "svg"` adds
|
||||||
|
`.doc-image-img--vector` (`width: 100%` + `object-fit: contain`) instead
|
||||||
|
of letting the drawing sit in the browser's 300×150 default box; the
|
||||||
|
chat figure and the Sources thumbnail keep their fixed caps (96px /
|
||||||
|
48px) and the modal is the full-size preview. Question images (the
|
||||||
|
paperclip) stay raster-only: an `.svg` upload is a 422.
|
||||||
|
|
||||||
|
Those bytes are served **sandboxed**: because the
|
||||||
|
`/api/documents/<id>/image` URL is directly navigable (not just an
|
||||||
|
`<img>` source, which can never run SVG script), an SVG response carries
|
||||||
|
`Content-Security-Policy: default-src 'none'; sandbox` so a stored
|
||||||
|
drawing can never execute script or act with the app's origin — raster
|
||||||
|
image responses keep their headers exactly as they were.
|
||||||
|
|
||||||
### Question images
|
### Question images
|
||||||
|
|
||||||
@@ -873,6 +913,12 @@ image:
|
|||||||
to `POST /api/chat-images` (capped at `BOR_CHAT_IMAGE_MAX_MB`,
|
to `POST /api/chat-images` (capped at `BOR_CHAT_IMAGE_MAX_MB`,
|
||||||
default 10 MiB — over is a 413) and stored under `BOR_CHAT_IMAGE_DIR`.
|
default 10 MiB — over is a 413) and stored under `BOR_CHAT_IMAGE_DIR`.
|
||||||
Saved and shared chats carry the returned *path* — never base64.
|
Saved and shared chats carry the returned *path* — never base64.
|
||||||
|
- **Raster only (phase 181).** The accepted upload extensions are the
|
||||||
|
**raster** members of `BOR_IMAGE_EXTENSIONS` — an `.svg` is a 422. The
|
||||||
|
vector family belongs to indexing, not to question attachments: a
|
||||||
|
vision endpoint rejects `image/svg+xml`, and the question-image serve
|
||||||
|
route deliberately never hands out a `.svg`, so accepting one would
|
||||||
|
store an unanswerable, unservable file.
|
||||||
- **Visible while uploading.** While the upload is in flight the
|
- **Visible while uploading.** While the upload is in flight the
|
||||||
preview strip shows a spinner + *"Uploading image…"* (its remove ✕
|
preview strip shows a spinner + *"Uploading image…"* (its remove ✕
|
||||||
disabled) and the Send button is disabled with the *"Uploading…"*
|
disabled) and the Send button is disabled with the *"Uploading…"*
|
||||||
@@ -1068,6 +1114,16 @@ Outermost to innermost, the stack is security-headers → gzip (only when
|
|||||||
security headers land on every response and `bytes=` in the budget line is the
|
security headers land on every response and `bytes=` in the budget line is the
|
||||||
route's own, uncompressed payload (see *Operational notes*):
|
route's own, uncompressed payload (see *Operational notes*):
|
||||||
|
|
||||||
|
> **Security headers (A20).** Every response carries the app-wide
|
||||||
|
> `Content-Security-Policy` (`default-src 'self'` plus the phase-123
|
||||||
|
> `img-src 'self' data:` carve-out), `X-Frame-Options: DENY` and
|
||||||
|
> `X-Content-Type-Options: nosniff` — and the middleware *preserves* a
|
||||||
|
> CSP an inner layer already set. One route uses that deliberately: SVG
|
||||||
|
> document bytes (phase 181) ship `default-src 'none'; sandbox` instead,
|
||||||
|
> because a directly-navigated stored `.svg` must never become a
|
||||||
|
> script-capable same-origin document. Raster/PDF image responses gain
|
||||||
|
> nothing — their header set is unchanged.
|
||||||
|
|
||||||
- **HTML pages** ship `Cache-Control: no-cache`, **no `etag`**, **no
|
- **HTML pages** ship `Cache-Control: no-cache`, **no `etag`**, **no
|
||||||
`last-modified`** — each visit always gets a fresh 200 body.
|
`last-modified`** — each visit always gets a fresh 200 body.
|
||||||
- **Assets are versioned and cached for a year.** Pages reference their
|
- **Assets are versioned and cached for a year.** Pages reference their
|
||||||
@@ -1299,12 +1355,12 @@ startup (the DB rows shadow the env, `.env` is never rewritten).
|
|||||||
| `BOR_UPLOAD_MAX_MB` | `512` | Cap (MiB) for uploaded source archives (zip-bomb guard) |
|
| `BOR_UPLOAD_MAX_MB` | `512` | Cap (MiB) for uploaded source archives (zip-bomb guard) |
|
||||||
| `BOR_SUGGESTIONS` | built-in list | JSON seed for onboarding chips (shown before the first saved question) |
|
| `BOR_SUGGESTIONS` | built-in list | JSON seed for onboarding chips (shown before the first saved question) |
|
||||||
|
|
||||||
### Image support (phases 122/123/124)
|
### Image support (phases 122/123/124/181)
|
||||||
|
|
||||||
| Env | Default | Meaning |
|
| Env | Default | Meaning |
|
||||||
|-----|---------|---------|
|
|-----|---------|---------|
|
||||||
| `BOR_IMAGES` | `0` | Master switch for image support — **off by default; enable only when `BOR_LLM_CHAT_MODEL` supports vision**. Gates both image documents and question images |
|
| `BOR_IMAGES` | `0` | Master switch for image support — **off by default; enable only when `BOR_LLM_CHAT_MODEL` supports vision**. Gates both image documents and question images |
|
||||||
| `BOR_IMAGE_EXTENSIONS` | `png,jpg,jpeg,webp,gif,bmp` | Extensions indexed as image documents (case-insensitive; separate from `BOR_IMPORT_EXTENSIONS`) |
|
| `BOR_IMAGE_EXTENSIONS` | `png,jpg,jpeg,webp,gif,bmp,svg` | Extensions indexed as image documents (case-insensitive; separate from `BOR_IMPORT_EXTENSIONS`). `svg` is the one **vector** member: indexed from its own `<title>`/`<desc>`/`<text>` with **no vision call**, served as `image/svg+xml` for a real preview |
|
||||||
| `BOR_IMAGE_DIR` | `~/bor-sources/images` | Persistent home for the served image bytes (uploads are replaced, checkouts re-cloned) |
|
| `BOR_IMAGE_DIR` | `~/bor-sources/images` | Persistent home for the served image bytes (uploads are replaced, checkouts re-cloned) |
|
||||||
| `BOR_CHAT_IMAGE_DIR` | `~/bor-sources/chat-images` | Where question images are stored (saved/shared chats carry the path, never base64) |
|
| `BOR_CHAT_IMAGE_DIR` | `~/bor-sources/chat-images` | Where question images are stored (saved/shared chats carry the path, never base64) |
|
||||||
| `BOR_CHAT_IMAGE_MAX_MB` | `10` | Question-image upload cap (MiB; must be > 0) |
|
| `BOR_CHAT_IMAGE_MAX_MB` | `10` | Question-image upload cap (MiB; must be > 0) |
|
||||||
@@ -1539,6 +1595,13 @@ instead of reaching the app.
|
|||||||
off (the default). Set `BOR_IMAGES=1` in `.env` and restart — only
|
off (the default). Set `BOR_IMAGES=1` in `.env` and restart — only
|
||||||
when `BOR_LLM_CHAT_MODEL` supports vision. Turning the toggle off
|
when `BOR_LLM_CHAT_MODEL` supports vision. Turning the toggle off
|
||||||
never destroys existing image documents (syncs prune-guard them).
|
never destroys existing image documents (syncs prune-guard them).
|
||||||
|
- **An `.svg` shows raw XML instead of the drawing** — image support is
|
||||||
|
off, so the file is a plain **text** document (phase 181). Set
|
||||||
|
`BOR_IMAGES=1` and re-sync: the next run converts the row in place
|
||||||
|
(bytes copied, `image_url` live, no re-import by hand). An `.svg` the
|
||||||
|
extractor refuses — over 2 MiB, carrying a `<!DOCTYPE`/`<!ENTITY`
|
||||||
|
declaration, or malformed XML — stays a text document and is counted
|
||||||
|
in `images_failed`.
|
||||||
- **Honest deflection** (the amber deflection bubble) — every question
|
- **Honest deflection** (the amber deflection bubble) — every question
|
||||||
passes the honesty gate: deflection happens
|
passes the honesty gate: deflection happens
|
||||||
when the best cosine similarity is below `BOR_RELEVANCE_THRESHOLD`
|
when the best cosine similarity is below `BOR_RELEVANCE_THRESHOLD`
|
||||||
|
|||||||
+26
-7
@@ -22,7 +22,9 @@ unguessable ``uuid4().hex`` — no enumeration value).
|
|||||||
The extension is the source of truth (the Content-Type header is a
|
The extension is the source of truth (the Content-Type header is a
|
||||||
hint — the archive-uploader precedent): it must be in the phase-122
|
hint — the archive-uploader precedent): it must be in the phase-122
|
||||||
image set (``settings.image_extension_set`` — the same frozenset the
|
image set (``settings.image_extension_set`` — the same frozenset the
|
||||||
image-document walk uses), lowercased; total bytes are streamed with a
|
image-document walk uses) minus the vector members
|
||||||
|
(:data:`_VECTOR_IMAGE_EXTENSIONS` — a question image is RASTER-only,
|
||||||
|
phase 181), lowercased; total bytes are streamed with a
|
||||||
``chat_image_max_mb`` cap (413, fixed detail naming the cap — never
|
``chat_image_max_mb`` cap (413, fixed detail naming the cap — never
|
||||||
echoing the filename).
|
echoing the filename).
|
||||||
"""
|
"""
|
||||||
@@ -55,6 +57,20 @@ _STREAM_CHUNK = 1 << 20
|
|||||||
#: upload endpoint's own naming).
|
#: upload endpoint's own naming).
|
||||||
_CHAT_IMAGE_FILENAME_RE = re.compile(r"^[0-9a-fA-F]{32}\.(png|jpe?g|webp|gif|bmp)$")
|
_CHAT_IMAGE_FILENAME_RE = re.compile(r"^[0-9a-fA-F]{32}\.(png|jpe?g|webp|gif|bmp)$")
|
||||||
|
|
||||||
|
#: The VECTOR members of ``settings.image_extension_set`` — excluded
|
||||||
|
#: from the question-image accept set (phase 181: the vector family
|
||||||
|
#: belongs to INDEXING, not to question attachments). Two reasons, both
|
||||||
|
#: hard: an OpenAI-compatible vision endpoint rejects
|
||||||
|
#: ``image/svg+xml`` (the phase-181 LOCKED A1 reason — the vector file
|
||||||
|
#: never reaches the describer at all), and
|
||||||
|
#: :data:`_CHAT_IMAGE_FILENAME_RE` above deliberately refuses to SERVE a
|
||||||
|
#: ``.svg``, so an accepted upload would be stored, unanswerable and
|
||||||
|
#: unservable (a broken figure in the user's own bubble). Subtracting
|
||||||
|
#: instead of touching the serve regex keeps question attachments
|
||||||
|
#: raster-only by design while ``svg`` rides the image set for the
|
||||||
|
#: document walk.
|
||||||
|
_VECTOR_IMAGE_EXTENSIONS: frozenset[str] = frozenset({".svg"})
|
||||||
|
|
||||||
|
|
||||||
@router.post("/chat-images")
|
@router.post("/chat-images")
|
||||||
async def upload_chat_image(
|
async def upload_chat_image(
|
||||||
@@ -66,10 +82,12 @@ async def upload_chat_image(
|
|||||||
Gates, in order:
|
Gates, in order:
|
||||||
|
|
||||||
1. **extension** — the file name's extension (lowercased) must be in
|
1. **extension** — the file name's extension (lowercased) must be in
|
||||||
the phase-122 image set (``settings.image_extension_set``); the
|
the phase-122 image set (``settings.image_extension_set``)
|
||||||
Content-Type header is a hint, never a source of truth (the
|
MINUS the vector members (:data:`_VECTOR_IMAGE_EXTENSIONS` —
|
||||||
archive-uploader precedent). A missing/unknown extension is a
|
question images are RASTER-only, phase 181); the Content-Type
|
||||||
422 naming the accepted set — a fixed detail, no filename echo.
|
header is a hint, never a source of truth (the archive-uploader
|
||||||
|
precedent). A missing/unknown/vector extension is a 422 naming
|
||||||
|
the accepted set — a fixed detail, no filename echo.
|
||||||
2. **size** — the bytes are streamed (1 MiB chunks) with the
|
2. **size** — the bytes are streamed (1 MiB chunks) with the
|
||||||
``chat_image_max_mb`` cap; over-cap is a 413 naming the cap
|
``chat_image_max_mb`` cap; over-cap is a 413 naming the cap
|
||||||
(fixed detail — never echoing the filename), the temp file is
|
(fixed detail — never echoing the filename), the temp file is
|
||||||
@@ -85,8 +103,9 @@ async def upload_chat_image(
|
|||||||
settings = get_settings()
|
settings = get_settings()
|
||||||
filename = file.filename or ""
|
filename = file.filename or ""
|
||||||
ext = Path(filename).suffix.lower()
|
ext = Path(filename).suffix.lower()
|
||||||
if ext not in settings.image_extension_set:
|
accepted_extensions = settings.image_extension_set - _VECTOR_IMAGE_EXTENSIONS
|
||||||
accepted = ", ".join(sorted(settings.image_extension_set))
|
if ext not in accepted_extensions:
|
||||||
|
accepted = ", ".join(sorted(accepted_extensions))
|
||||||
raise HTTPException(
|
raise HTTPException(
|
||||||
status_code=422, detail=f"only {accepted} images are accepted"
|
status_code=422, detail=f"only {accepted} images are accepted"
|
||||||
)
|
)
|
||||||
|
|||||||
+59
-5
@@ -52,10 +52,15 @@ BYTES route: the persistent copy behind an ``is_image`` document's
|
|||||||
``image_path``, served with the extension's ``Content-Type`` (the
|
``image_path``, served with the extension's ``Content-Type`` (the
|
||||||
``app.rag.summarizer.IMAGE_MIMES`` map — one map, one truth) and a
|
``app.rag.summarizer.IMAGE_MIMES`` map — one map, one truth) and a
|
||||||
``Cache-Control: private, max-age=3600`` header (the bytes are
|
``Cache-Control: private, max-age=3600`` header (the bytes are
|
||||||
content-hashed — long enough, bustable by re-upload). PUBLIC (phase
|
content-hashed — long enough, bustable by re-upload), plus — for the
|
||||||
124 — the phase-123 chat-image precedent: the image is part of a
|
vector ``image/svg+xml`` entry only, phase 181 LOCKED A3 — a
|
||||||
shared chat's rendered content, and the shared chat — the app's ONLY
|
``Content-Security-Policy: default-src 'none'; sandbox`` so a DIRECT
|
||||||
anonymous surface — must render the answer figure faithfully for
|
navigation to the URL cannot run script from stored SVG markup (the
|
||||||
|
``SecurityHeadersMiddleware`` preserves that inner CSP; raster
|
||||||
|
responses gain nothing). PUBLIC (phase 124 — the phase-123 chat-image
|
||||||
|
precedent: the image is part of a shared chat's rendered content, and
|
||||||
|
the shared chat — the app's ONLY anonymous surface — must render the
|
||||||
|
answer figure faithfully for
|
||||||
guests; the unguessable ``uuid4`` document id IS the credential, no
|
guests; the unguessable ``uuid4`` document id IS the credential, no
|
||||||
enumeration value): a missing doc, a non-image doc, a doc whose
|
enumeration value): a missing doc, a non-image doc, a doc whose
|
||||||
``image_path`` is NULL, or a row whose copy was lost all map to 404
|
``image_path`` is NULL, or a row whose copy was lost all map to 404
|
||||||
@@ -132,6 +137,23 @@ from app.schemas import (
|
|||||||
|
|
||||||
router = APIRouter(tags=["kb"])
|
router = APIRouter(tags=["kb"])
|
||||||
|
|
||||||
|
#: The vector media type (phase 181, task 03) — read out of
|
||||||
|
#: :data:`app.rag.summarizer.IMAGE_MIMES` rather than restated, so the
|
||||||
|
#: guard below can never drift from the ``Content-Type`` the route
|
||||||
|
#: sends for an ``.svg`` (one map, one truth).
|
||||||
|
SVG_MEDIA_TYPE = IMAGE_MIMES[".svg"]
|
||||||
|
|
||||||
|
#: The CSP an SVG bytes response carries (phase 181, task 03, LOCKED
|
||||||
|
#: A3). ``sandbox`` — no scripting, and the document gets an OPAQUE
|
||||||
|
#: origin (no same-origin fetch/cookie/form capability);
|
||||||
|
#: ``default-src 'none'`` — no subresource of any kind. Scoped to the
|
||||||
|
#: vector type ON PURPOSE: an ``<img>`` never runs SVG script, but this
|
||||||
|
#: URL is directly navigable, and the bytes come from an untrusted
|
||||||
|
#: source tree, so a direct visit must not become a script-capable
|
||||||
|
#: same-origin document. Raster types are untouched (their header set
|
||||||
|
#: stays byte-identical). See :func:`get_document_image`.
|
||||||
|
SVG_SANDBOX_CSP = "default-src 'none'; sandbox"
|
||||||
|
|
||||||
|
|
||||||
def doc_format(path: str, extensions: frozenset[str] = frozenset()) -> str:
|
def doc_format(path: str, extensions: frozenset[str] = frozenset()) -> str:
|
||||||
"""Lowercased path suffix without its dot (``kubernetes.md`` → ``md``,
|
"""Lowercased path suffix without its dot (``kubernetes.md`` → ``md``,
|
||||||
@@ -353,6 +375,33 @@ def get_document_image(
|
|||||||
``application/octet-stream``) and ``Cache-Control: private,
|
``application/octet-stream``) and ``Cache-Control: private,
|
||||||
max-age=3600`` (the bytes are content-hashed — long enough to be
|
max-age=3600`` (the bytes are content-hashed — long enough to be
|
||||||
useful, bustable by re-upload).
|
useful, bustable by re-upload).
|
||||||
|
|
||||||
|
Phase 181 (task 03, LOCKED A3): an ``image/svg+xml`` response — the
|
||||||
|
vector entry of that same map, derived from the ROW's extension and
|
||||||
|
NEVER from a client value — carries ONE extra header,
|
||||||
|
:data:`SVG_SANDBOX_CSP` (``default-src 'none'; sandbox``). An
|
||||||
|
``<img>`` can never execute an SVG's script, which is why the
|
||||||
|
figure surfaces are safe as they stand; but this URL is also
|
||||||
|
DIRECTLY NAVIGABLE (a guest who pastes it into the address bar),
|
||||||
|
and without the policy a stored SVG would render as a
|
||||||
|
script-capable SAME-ORIGIN document — the ingestion path takes
|
||||||
|
untrusted files from a source tree, so that is a stored-XSS seam
|
||||||
|
scoped to the app's origin. ``sandbox`` blocks scripting AND makes
|
||||||
|
the response an opaque origin (no same-origin fetches, cookies or
|
||||||
|
form submission from the document); ``default-src 'none'`` blocks
|
||||||
|
every subresource load, so the SVG cannot pull a tracker or a
|
||||||
|
remote image either. Deliberately NOT applied to the raster types:
|
||||||
|
their header set stays byte-identical (a PNG needs no policy
|
||||||
|
change, and adding one would risk the phase-123/124 image
|
||||||
|
surfaces), and no ``Content-Disposition: attachment`` is set — the
|
||||||
|
graphic must render inline, and ``sandbox`` is the stronger, less
|
||||||
|
intrusive control. The route sets the value on the response
|
||||||
|
itself; :class:`app.core.security_headers.SecurityHeadersMiddleware`
|
||||||
|
PRESERVES a CSP an inner layer already set, so this is the policy
|
||||||
|
the browser sees (the app-wide ``CSP`` string would be wrong here —
|
||||||
|
``default-src 'self'`` would grant the SVG document the app's own
|
||||||
|
origins), while the middleware's global
|
||||||
|
``X-Content-Type-Options: nosniff`` still applies.
|
||||||
"""
|
"""
|
||||||
try:
|
try:
|
||||||
uid = uuid.UUID(doc_id)
|
uid = uuid.UUID(doc_id)
|
||||||
@@ -368,10 +417,15 @@ def get_document_image(
|
|||||||
# onerror fallback renders the "image unavailable" note.
|
# onerror fallback renders the "image unavailable" note.
|
||||||
raise HTTPException(status_code=404, detail="document not found")
|
raise HTTPException(status_code=404, detail="document not found")
|
||||||
media_type = IMAGE_MIMES.get(image_file.suffix.lower(), IMAGE_FALLBACK_MIME)
|
media_type = IMAGE_MIMES.get(image_file.suffix.lower(), IMAGE_FALLBACK_MIME)
|
||||||
|
headers = {"Cache-Control": "private, max-age=3600"}
|
||||||
|
if media_type == SVG_MEDIA_TYPE:
|
||||||
|
# Phase 181 (LOCKED A3): vector bytes only — see the route
|
||||||
|
# docstring. Raster responses keep the header set they had.
|
||||||
|
headers["Content-Security-Policy"] = SVG_SANDBOX_CSP
|
||||||
return FileResponse(
|
return FileResponse(
|
||||||
image_file,
|
image_file,
|
||||||
media_type=media_type,
|
media_type=media_type,
|
||||||
headers={"Cache-Control": "private, max-age=3600"},
|
headers=headers,
|
||||||
)
|
)
|
||||||
|
|
||||||
|
|
||||||
|
|||||||
+12
-1
@@ -433,7 +433,18 @@ class Settings(BaseSettings):
|
|||||||
#: knob). The validator rejects an empty list and malformed tokens,
|
#: knob). The validator rejects an empty list and malformed tokens,
|
||||||
#: exactly like ``import_extensions`` (a typo would otherwise index
|
#: exactly like ``import_extensions`` (a typo would otherwise index
|
||||||
#: zero images silently).
|
#: zero images silently).
|
||||||
image_extensions: str = "png,jpg,jpeg,webp,gif,bmp"
|
#:
|
||||||
|
#: ``svg`` (phase 181, LOCKED A1) is the one VECTOR member of the
|
||||||
|
#: family: an ``.svg`` takes the same image branch (bytes copied to
|
||||||
|
#: ``image_dir``, ``is_image``/``image_path``, served as
|
||||||
|
#: ``image/svg+xml``) but NEVER the vision call — its searchable
|
||||||
|
#: text comes from the file itself
|
||||||
|
#: (:func:`app.rag.svg_text.extract_svg_text`, the ``<title>`` /
|
||||||
|
#: ``<desc>`` / ``<text>`` words or an honest structural line).
|
||||||
|
#: ``images`` remains the SINGLE knob (LOCKED A7): with it off an
|
||||||
|
#: ``.svg`` stays a plain text document (raw XML in the viewer) and
|
||||||
|
#: the walk never admits it as an image — no special-casing.
|
||||||
|
image_extensions: str = "png,jpg,jpeg,webp,gif,bmp,svg"
|
||||||
#: Where ingested image bytes are copied for serving (phase 122,
|
#: Where ingested image bytes are copied for serving (phase 122,
|
||||||
#: ``BOR_IMAGE_DIR``). Raw string — ``Path.expanduser()`` is applied
|
#: ``BOR_IMAGE_DIR``). Raw string — ``Path.expanduser()`` is applied
|
||||||
#: by the importer, not here (the ``sources_dir``/``upload_dir``
|
#: by the importer, not here (the ``sources_dir``/``upload_dir``
|
||||||
|
|||||||
+121
-7
@@ -111,6 +111,20 @@ summary (stored verbatim, no ``lite`` call, no pointer line), so the
|
|||||||
``is_summary`` position −1 chunk mirrors ``Document.summary``, which
|
``is_summary`` position −1 chunk mirrors ``Document.summary``, which
|
||||||
equals ``Document.content``.
|
equals ``Document.content``.
|
||||||
|
|
||||||
|
The image branch is RASTER (vision-described, the paragraph above) OR
|
||||||
|
VECTOR (phase 181, LOCKED A1: an ``.svg``). A vector file takes the
|
||||||
|
SAME branch — bytes digest, ``image_dir`` copy, ``is_image``/``image_path``,
|
||||||
|
the ``summary = content`` convention, the phase-125 ``image_context``
|
||||||
|
enrichment — but NEVER the model call: its searchable text is extracted
|
||||||
|
from the file itself (:func:`app.rag.svg_text.extract_svg_text` — the
|
||||||
|
``<title>``/``<desc>``/``<text>`` words, or LOCKED A2's structural line
|
||||||
|
built from the stem, the shape census and the ``viewBox`` when the
|
||||||
|
drawing carries no words at all). A file the extractor refuses (over the
|
||||||
|
size cap, DOCTYPE/entity-bearing, malformed XML) fails soft exactly like
|
||||||
|
an undescribed raster: ``images_failed`` + a warning, no row, no copy.
|
||||||
|
Successful vector indexes count ``svg_indexed`` (log-only) and log one
|
||||||
|
``import: svg indexed`` line.
|
||||||
|
|
||||||
PDF files (phase 126, LOCKED A1/A2/A3/A4): a ``.pdf`` file (the
|
PDF files (phase 126, LOCKED A1/A2/A3/A4): a ``.pdf`` file (the
|
||||||
``pdf`` token of ``BOR_IMPORT_EXTENSIONS``) takes the PDF index path
|
``pdf`` token of ``BOR_IMPORT_EXTENSIONS``) takes the PDF index path
|
||||||
(:func:`_index_pdf_file` — the branch sits in :func:`_index_file`
|
(:func:`_index_pdf_file` — the branch sits in :func:`_index_file`
|
||||||
@@ -359,6 +373,7 @@ from app.rag.summarizer import (
|
|||||||
describe_pdf_page,
|
describe_pdf_page,
|
||||||
generate_summary,
|
generate_summary,
|
||||||
)
|
)
|
||||||
|
from app.rag.svg_text import SvgText, extract_svg_text
|
||||||
|
|
||||||
logger = logging.getLogger("app.importer")
|
logger = logging.getLogger("app.importer")
|
||||||
|
|
||||||
@@ -433,6 +448,14 @@ class ImportSummary:
|
|||||||
#: Fail-soft: the sync continues, this counter + the warning line
|
#: Fail-soft: the sync continues, this counter + the warning line
|
||||||
#: are the signal.
|
#: are the signal.
|
||||||
images_failed: int = 0
|
images_failed: int = 0
|
||||||
|
#: Vector image docs (phase 181, task 02 — LOCKED A1) indexed from
|
||||||
|
#: their OWN extracted text (:func:`app.rag.svg_text.extract_svg_text`)
|
||||||
|
#: with NO vision call. A subset of the image branch: it does NOT
|
||||||
|
#: move ``added``/``updated`` (those count as for any other image).
|
||||||
|
#: Log-only (the sync ``detail`` API shape is untouched — the stats
|
||||||
|
#: log-only convention, phase 133): the owner reads it in the sync
|
||||||
|
#: log beside the per-file ``import: svg indexed`` line.
|
||||||
|
svg_indexed: int = 0
|
||||||
#: PDFs (phase 133, task 03) whose text layer failed the garbage
|
#: PDFs (phase 133, task 03) whose text layer failed the garbage
|
||||||
#: gate (:func:`app.rag.pdf.is_usable_pdf_text` — empty layer or
|
#: gate (:func:`app.rag.pdf.is_usable_pdf_text` — empty layer or
|
||||||
#: XML/binary soup) and took the page-screenshot vision fallback
|
#: XML/binary soup) and took the page-screenshot vision fallback
|
||||||
@@ -498,7 +521,7 @@ class ImportSummary:
|
|||||||
logger.info(
|
logger.info(
|
||||||
"import: summary files=%d added=%d updated=%d unchanged=%d pruned=%d "
|
"import: summary files=%d added=%d updated=%d unchanged=%d pruned=%d "
|
||||||
"errors=%d chunks=%d embed_batches=%d summaries=%d summary_errors=%d "
|
"errors=%d chunks=%d embed_batches=%d summaries=%d summary_errors=%d "
|
||||||
"summary_backfilled=%d dates_updated=%d images_failed=%d "
|
"summary_backfilled=%d dates_updated=%d images_failed=%d svg_indexed=%d "
|
||||||
"image_context_backfilled=%d pdfs_screenshotted=%d "
|
"image_context_backfilled=%d pdfs_screenshotted=%d "
|
||||||
"pdfs_pictorial=%d pdf_pictorial_pages=%d pdf_images_deduped=%d "
|
"pdfs_pictorial=%d pdf_pictorial_pages=%d pdf_images_deduped=%d "
|
||||||
"pdf_images_blank_skipped=%d formats=%s",
|
"pdf_images_blank_skipped=%d formats=%s",
|
||||||
@@ -515,6 +538,7 @@ class ImportSummary:
|
|||||||
self.summary_backfilled,
|
self.summary_backfilled,
|
||||||
self.dates_updated,
|
self.dates_updated,
|
||||||
self.images_failed,
|
self.images_failed,
|
||||||
|
self.svg_indexed,
|
||||||
self.image_context_backfilled,
|
self.image_context_backfilled,
|
||||||
self.pdfs_screenshotted,
|
self.pdfs_screenshotted,
|
||||||
self.pdfs_pictorial,
|
self.pdfs_pictorial,
|
||||||
@@ -1325,6 +1349,38 @@ def _regate_pdf_strip(
|
|||||||
)
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _row_kind_mismatch(path: Path, doc: Document, settings: Settings) -> bool:
|
||||||
|
"""Phase 181 (task 02, design D4): does this row's KIND disagree with
|
||||||
|
how *path* would be indexed NOW — the ONE case that must fall
|
||||||
|
THROUGH the phase-147 unchanged fast path?
|
||||||
|
|
||||||
|
``True`` only while the ``images`` toggle is ON and the file matches
|
||||||
|
the image extension set while the row is still a TEXT row
|
||||||
|
(``is_image`` False). That is exactly the migration this phase needs:
|
||||||
|
an ``.svg`` imported before ``svg`` joined
|
||||||
|
:py:attr:`Settings.image_extension_set` is a text document whose
|
||||||
|
``content`` is the raw XML — and because the text digest of an ASCII
|
||||||
|
file equals the digest of its bytes, its ``file_hash`` (and even its
|
||||||
|
``content_hash``) matches perfectly, so a hash-only fast path would
|
||||||
|
freeze it as XML forever: no bytes route, no preview, no
|
||||||
|
``image_url`` citation figure — the exact complaint this phase
|
||||||
|
answers. Falling through runs the image branch, which rewrites the
|
||||||
|
row (``is_image``/``image_path``/extracted ``content`` + the
|
||||||
|
``image_dir`` copy) through the normal index write: no alembic
|
||||||
|
migration, no backfill script, and the next sync is fast again.
|
||||||
|
|
||||||
|
The REVERSE direction is deliberately NOT forced: with the toggle
|
||||||
|
OFF the walk never admits an image file (so an ``is_image`` row never
|
||||||
|
reaches :func:`_index_file` at all) and the phase-122 prune guard
|
||||||
|
keeps the row; requiring ``settings.images`` here keeps that
|
||||||
|
contract byte-identical (``svg`` in ``BOR_IMPORT_EXTENSIONS`` with
|
||||||
|
the toggle off still means "text document", LOCKED A7).
|
||||||
|
"""
|
||||||
|
if not settings.images or doc.is_image:
|
||||||
|
return False
|
||||||
|
return match_extension(path, settings.image_extension_set) is not None
|
||||||
|
|
||||||
|
|
||||||
async def _index_file(
|
async def _index_file(
|
||||||
session: Session,
|
session: Session,
|
||||||
*,
|
*,
|
||||||
@@ -1400,10 +1456,17 @@ async def _index_file(
|
|||||||
# fast path (and the branches' unchanged path) refresh the
|
# fast path (and the branches' unchanged path) refresh the
|
||||||
# stored date from the same source.
|
# stored date from the same source.
|
||||||
raw_date = file_mtime_datetime(full_path)
|
raw_date = file_mtime_datetime(full_path)
|
||||||
if doc is not None and doc.file_hash == file_digest:
|
if (
|
||||||
|
doc is not None
|
||||||
|
and doc.file_hash == file_digest
|
||||||
|
and not _row_kind_mismatch(full_path, doc, settings)
|
||||||
|
):
|
||||||
# The fast path: the raw bytes are what was indexed last time
|
# The fast path: the raw bytes are what was indexed last time
|
||||||
# — the bookkeeping (backfill / D1 / D4) runs in the shared
|
# — the bookkeeping (backfill / D1 / D4) runs in the shared
|
||||||
# helper, nothing else does.
|
# helper, nothing else does. Phase 181 (task 02, D4): "the same
|
||||||
|
# bytes" is only "the same document" when the row's KIND also
|
||||||
|
# matches how the file would be indexed NOW — see
|
||||||
|
# :func:`_row_kind_mismatch` (the SVG text→image migration).
|
||||||
await _unchanged_doc(
|
await _unchanged_doc(
|
||||||
session, doc, source=source, rel=rel, llm=llm, summary=summary,
|
session, doc, source=source, rel=rel, llm=llm, summary=summary,
|
||||||
raw_date=raw_date, file_digest=file_digest, full_path=full_path,
|
raw_date=raw_date, file_digest=file_digest, full_path=full_path,
|
||||||
@@ -3542,7 +3605,17 @@ async def _index_image_file(
|
|||||||
on_chunk: Callable[[int, int], None] | None = None,
|
on_chunk: Callable[[int, int], None] | None = None,
|
||||||
) -> None:
|
) -> None:
|
||||||
"""The phase-122 image branch of :func:`_index_file` — a standalone
|
"""The phase-122 image branch of :func:`_index_file` — a standalone
|
||||||
image is indexed from its BYTES, never its text:
|
image is indexed from its BYTES, never its text.
|
||||||
|
|
||||||
|
Phase 181 (task 02, LOCKED A1) makes the branch RASTER **or**
|
||||||
|
VECTOR: an ``.svg`` takes this same path (digest, copy, ``is_image``,
|
||||||
|
``image_path``, chunking, ``summary = content``, the phase-125
|
||||||
|
``image_context`` enrichment) but its ``content`` comes from
|
||||||
|
:func:`app.rag.svg_text.extract_svg_text` instead of the vision
|
||||||
|
seam, and a success counts the log-only ``svg_indexed``. The bullets
|
||||||
|
and the fail-soft below are shared by both kinds — the only
|
||||||
|
difference is where the text comes from.
|
||||||
|
|
||||||
|
|
||||||
* the sha256 digest is over the raw bytes (the digest rule is
|
* the sha256 digest is over the raw bytes (the digest rule is
|
||||||
content identity — the same bytes are the same document);
|
content identity — the same bytes are the same document);
|
||||||
@@ -3600,13 +3673,20 @@ async def _index_image_file(
|
|||||||
digest = hashlib.sha256(data).hexdigest()
|
digest = hashlib.sha256(data).hexdigest()
|
||||||
doc = session.scalar(select(Document).where(Document.source == source, Document.path == rel))
|
doc = session.scalar(select(Document).where(Document.source == source, Document.path == rel))
|
||||||
|
|
||||||
if doc is not None and doc.content_hash == digest:
|
if doc is not None and doc.is_image and doc.content_hash == digest:
|
||||||
# Phase 147: the shared unchanged path — a pre-phase NULL-hash
|
# Phase 147: the shared unchanged path — a pre-phase NULL-hash
|
||||||
# row, or a bytes-changed/content-same image (the byte digest
|
# row, or a bytes-changed/content-same image (the byte digest
|
||||||
# still matches). The helper stamps the new file hash (the
|
# still matches). The helper stamps the new file hash (the
|
||||||
# next sync takes the fast path); for an image the summary is
|
# next sync takes the fast path); for an image the summary is
|
||||||
# the stored description (``doc.content``), so the image-aware
|
# the stored description (``doc.content``), so the image-aware
|
||||||
# ``_store_summary`` backfill re-stores it verbatim.
|
# ``_store_summary`` backfill re-stores it verbatim.
|
||||||
|
# Phase 181 (task 02, D4): ``doc.is_image`` joins the guard — a
|
||||||
|
# NON-image row routed here by :func:`_row_kind_mismatch` is the
|
||||||
|
# migration case, not an unchanged one. An ASCII SVG indexed as
|
||||||
|
# TEXT digests to the SAME value here (the text path hashes
|
||||||
|
# ``content.encode()``, which for a byte-clean file equals the
|
||||||
|
# raw-bytes digest), so without it the text→image conversion
|
||||||
|
# would report ``unchanged`` forever and never produce the copy.
|
||||||
await _unchanged_doc(
|
await _unchanged_doc(
|
||||||
session, doc, source=source, rel=rel, llm=llm, summary=summary,
|
session, doc, source=source, rel=rel, llm=llm, summary=summary,
|
||||||
raw_date=raw_date, file_digest=file_digest, full_path=full_path,
|
raw_date=raw_date, file_digest=file_digest, full_path=full_path,
|
||||||
@@ -3629,16 +3709,50 @@ async def _index_image_file(
|
|||||||
# it is generated BEFORE anything is written, so a failure skips the
|
# it is generated BEFORE anything is written, so a failure skips the
|
||||||
# doc with no row and no copy (the copy is only made after a
|
# doc with no row and no copy (the copy is only made after a
|
||||||
# successful description — a failure never leaves an orphan).
|
# successful description — a failure never leaves an orphan).
|
||||||
|
#
|
||||||
|
# Phase 181 (task 02, LOCKED A1): the VECTOR branch. An ``.svg`` is
|
||||||
|
# described by NOBODY — OpenAI-compatible vision endpoints reject
|
||||||
|
# ``image/svg+xml`` and no rasteriser dependency is in scope — so
|
||||||
|
# its text comes out of the file itself (its ``<title>``/``<desc>``/
|
||||||
|
# ``<text>`` words, or LOCKED A2's structural line). Everything else
|
||||||
|
# in this function is shared: the byte digest, the ``image_dir``
|
||||||
|
# copy, ``is_image``/``image_path``, the title, the counters, the
|
||||||
|
# phase-125 ``image_context`` prepend below, the chunk pipeline and
|
||||||
|
# ``summary = content``. The seam below is not reached, so no vision
|
||||||
|
# call happens for a vector file (pinned by the unit tests).
|
||||||
|
is_vector = full_path.suffix.lower() == ".svg"
|
||||||
|
svg: SvgText | None = None
|
||||||
|
if is_vector:
|
||||||
|
svg = extract_svg_text(data, stem=full_path.stem)
|
||||||
|
content = None if svg is None else svg.text
|
||||||
|
else:
|
||||||
content = await _describe_or_skip(
|
content = await _describe_or_skip(
|
||||||
llm, data=data, source=source, rel=rel, full_path=full_path,
|
llm, data=data, source=source, rel=rel, full_path=full_path,
|
||||||
context=image_context,
|
context=image_context,
|
||||||
)
|
)
|
||||||
if content is None:
|
if content is None:
|
||||||
# LOCKED A3 fail-soft: an undescribed image is unsearchable
|
# LOCKED A3 fail-soft: an undescribed image is unsearchable
|
||||||
# noise — skip the doc entirely (no row, no copy).
|
# noise — skip the doc entirely (no row, no copy). Phase 181
|
||||||
|
# (LOCKED A2): the vector case skips ONLY when the extractor
|
||||||
|
# refused the file (over the size cap, DOCTYPE/entity-bearing,
|
||||||
|
# malformed XML) — a textless SVG is NOT a failure, it gets the
|
||||||
|
# structural line.
|
||||||
summary.images_failed += 1
|
summary.images_failed += 1
|
||||||
logger.warning("import: image description failed source=%s path=%s", source, rel)
|
logger.warning(
|
||||||
|
"import: %s source=%s path=%s",
|
||||||
|
"svg text extraction failed" if is_vector else "image description failed",
|
||||||
|
source, rel,
|
||||||
|
)
|
||||||
return
|
return
|
||||||
|
if svg is not None:
|
||||||
|
# Phase 181 (task 02): the log-only vector counter + the
|
||||||
|
# auditability line (AGENTS rule 10) — ``structural=True`` marks
|
||||||
|
# the docs whose text is the LOCKED A2 fallback line.
|
||||||
|
summary.svg_indexed += 1
|
||||||
|
logger.info(
|
||||||
|
"import: svg indexed source=%s path=%s structural=%s chars=%d",
|
||||||
|
source, rel, svg.structural, len(svg.text),
|
||||||
|
)
|
||||||
if image_context is not None:
|
if image_context is not None:
|
||||||
# Phase 125 (task 03): the enriched content — the reference
|
# Phase 125 (task 03): the enriched content — the reference
|
||||||
# context FIRST (for retrieval), then the description. The
|
# context FIRST (for retrieval), then the description. The
|
||||||
|
|||||||
@@ -149,6 +149,17 @@ PICTORIAL_PROMPT = f"{PICTORIAL_PAGE_MODE}: {PICTORIAL_INSTRUCTION}"
|
|||||||
#: takes the :data:`IMAGE_FALLBACK_MIME` data-URL mime in the describe
|
#: takes the :data:`IMAGE_FALLBACK_MIME` data-URL mime in the describe
|
||||||
#: call — the vision endpoint may reject it, and the fail-soft skip
|
#: call — the vision endpoint may reject it, and the fail-soft skip
|
||||||
#: (``images_failed``) is the honest outcome.
|
#: (``images_failed``) is the honest outcome.
|
||||||
|
#:
|
||||||
|
#: Phase 181 adds ``.svg`` — the ONE **vector** entry in the map. It
|
||||||
|
#: exists for the BYTES route only: the vector import branch
|
||||||
|
#: (``app/rag/importer.py::_index_image_file``) takes
|
||||||
|
#: :func:`app.rag.svg_text.extract_svg_text` INSTEAD of the vision
|
||||||
|
#: describe call (LOCKED A1), so an ``.svg`` never reaches
|
||||||
|
#: :func:`image_data_url` and this value is never put in a multimodal
|
||||||
|
#: message. ``app/api/docs.py::get_document_image`` reads the same map
|
||||||
|
#: for the ``Content-Type`` (one map, one truth), and that route is
|
||||||
|
#: where the vector value earns its place — plus the sandboxed CSP the
|
||||||
|
#: vector bytes need and the rasters do not (LOCKED A3, phase 181).
|
||||||
IMAGE_MIMES: dict[str, str] = {
|
IMAGE_MIMES: dict[str, str] = {
|
||||||
".png": "image/png",
|
".png": "image/png",
|
||||||
".jpg": "image/jpeg",
|
".jpg": "image/jpeg",
|
||||||
@@ -156,6 +167,7 @@ IMAGE_MIMES: dict[str, str] = {
|
|||||||
".webp": "image/webp",
|
".webp": "image/webp",
|
||||||
".gif": "image/gif",
|
".gif": "image/gif",
|
||||||
".bmp": "image/bmp",
|
".bmp": "image/bmp",
|
||||||
|
".svg": "image/svg+xml",
|
||||||
}
|
}
|
||||||
|
|
||||||
#: The data-URL mime for an image extension :data:`IMAGE_MIMES` does not
|
#: The data-URL mime for an image extension :data:`IMAGE_MIMES` does not
|
||||||
|
|||||||
@@ -0,0 +1,293 @@
|
|||||||
|
"""SVG text extraction — the words a vector drawing carries (phase 181).
|
||||||
|
|
||||||
|
Pure functions, no I/O — fully unit-testable. Stdlib only (the
|
||||||
|
:mod:`app.rag.md_images` precedent, phase 125).
|
||||||
|
|
||||||
|
An ``.svg`` joins the image family (LOCKED A1, phase 181): its BYTES are
|
||||||
|
served to the browser so every citation surface paints the graphic
|
||||||
|
instead of XML. But OpenAI-compatible vision endpoints reject
|
||||||
|
``image/svg+xml``, and adding a rasteriser dependency is out of scope —
|
||||||
|
so the vector branch never calls the describer. The text that makes the
|
||||||
|
document **retrievable** comes from the file itself:
|
||||||
|
|
||||||
|
* ``<title>``, ``<desc>`` and ``<text>`` (LOCKED D2) — collected in
|
||||||
|
DOCUMENT order by LOCAL name (namespace-agnostic: ``<title>`` and
|
||||||
|
``{http://www.w3.org/2000/svg}title`` both count, declared ``svg:``
|
||||||
|
prefixes included), whitespace-normalised, de-duplicated
|
||||||
|
case-insensitively keeping the FIRST occurrence, joined with ``\\n``.
|
||||||
|
``<tspan>`` children are joined WITH a space so ``a<tspan>b</tspan>``
|
||||||
|
reads "a b", not "ab". XML entity references inside the text
|
||||||
|
(``&``) are decoded by the parser — the correct direction, the
|
||||||
|
renderer re-escapes on display.
|
||||||
|
* a textless diagram (pure ``<path>``s — extremely common) still gets
|
||||||
|
honest, searchable text (LOCKED A2): one structural line built ONLY
|
||||||
|
from facts the file states, never invented prose.
|
||||||
|
|
||||||
|
Guards, applied BEFORE any parse (untrusted markup, the sibling of
|
||||||
|
LOCKED A3):
|
||||||
|
|
||||||
|
* empty / whitespace-only input → ``None``;
|
||||||
|
* more than :data:`MAX_SVG_BYTES` → ``None`` (memory guard);
|
||||||
|
* a ``<!DOCTYPE`` or ``<!ENTITY`` declaration anywhere in the bytes →
|
||||||
|
``None``. Refusing the doctype outright is what makes stdlib
|
||||||
|
:mod:`xml.etree.ElementTree` safe here: no external entities, no
|
||||||
|
entity expansion (the billion-laughs class);
|
||||||
|
* malformed XML (:class:`xml.etree.ElementTree.ParseError`) → ``None``
|
||||||
|
— never re-raised, the caller's fail-soft is a skip, not a crash.
|
||||||
|
|
||||||
|
What is deliberately NOT followed: ``<image>`` elements embedding a
|
||||||
|
raster inside the SVG (their ``href`` is not text), and
|
||||||
|
``aria-label``/``role`` attributes — ``<title>``/``<desc>`` are the
|
||||||
|
SVG-spec carriers of words and the fallback line covers the rest.
|
||||||
|
"""
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import math
|
||||||
|
import re
|
||||||
|
from dataclasses import dataclass
|
||||||
|
from xml.etree import ElementTree
|
||||||
|
|
||||||
|
#: Hard input cap for one SVG (phase 181, LOCKED A3's sibling) — bytes
|
||||||
|
#: over this are refused without a parse attempt, so a hostile or
|
||||||
|
#: accidentally gigantic file can never be the memory event of a sync.
|
||||||
|
MAX_SVG_BYTES: int = 2 * 1024 * 1024
|
||||||
|
|
||||||
|
#: Cap on the EXTRACTED text (phase 181). It binds on the result, never
|
||||||
|
#: silently mid-word: an over-cap extraction is cut at the last newline
|
||||||
|
#: or period before the cap (a hard cut only when the window holds no
|
||||||
|
#: boundary at all).
|
||||||
|
MAX_SVG_TEXT_CHARS: int = 4000
|
||||||
|
|
||||||
|
#: A document type declaration in the raw bytes (lower-cased marker).
|
||||||
|
#: Its presence — anywhere — refuses the input (billion-laughs /
|
||||||
|
#: external-entity class).
|
||||||
|
_DOCTYPE_MARKER = b"<!doctype"
|
||||||
|
|
||||||
|
#: An entity declaration in the raw bytes (lower-cased marker) — same
|
||||||
|
#: refusal as :data:`_DOCTYPE_MARKER`, checked separately so a stray
|
||||||
|
#: ``<!ENTITY`` inside a stray doctype-less prolog is still caught.
|
||||||
|
_ENTITY_MARKER = b"<!entity"
|
||||||
|
|
||||||
|
#: Elements whose whole text block is one item, nested markup
|
||||||
|
#: concatenated (:meth:`ElementTree.Element.itertext`).
|
||||||
|
_BLOCK_TEXT_NAMES = frozenset({"title", "desc"})
|
||||||
|
|
||||||
|
#: Elements whose text is joined with a space so inline children
|
||||||
|
#: (``<tspan>``) do not weld their neighbours together.
|
||||||
|
_SPACED_TEXT_NAMES = frozenset({"text"})
|
||||||
|
|
||||||
|
#: Local names that are structure, not drawing: never counted as shapes
|
||||||
|
#: in the fallback line (:func:`_shape_clause`).
|
||||||
|
_NON_SHAPE_NAMES = frozenset({"svg", "title", "desc"})
|
||||||
|
|
||||||
|
#: How many element kinds the fallback line names (LOCKED D3: "the
|
||||||
|
#: three most common element local names with counts").
|
||||||
|
_TOP_SHAPE_NAMES = 3
|
||||||
|
|
||||||
|
#: ``viewBox`` components: four numbers separated by whitespace and/or
|
||||||
|
#: commas (the SVG-spec grammar, which allows either).
|
||||||
|
_VIEWBOX_SPLIT_RE = re.compile(r"[\s,]+")
|
||||||
|
|
||||||
|
#: The multiplication sign used between a shape name and its count, and
|
||||||
|
#: between the viewBox width and height — the app's own typography.
|
||||||
|
_TIMES = "×"
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass(frozen=True)
|
||||||
|
class SvgText:
|
||||||
|
"""The searchable text of one SVG document (phase 181, LOCKED A1/A2).
|
||||||
|
|
||||||
|
* ``text`` — what gets stored as the document's ``content`` (and
|
||||||
|
``summary``, per the phase-122 image convention);
|
||||||
|
* ``structural`` — ``True`` when the document carried no words and
|
||||||
|
``text`` is the structural fallback line, so the importer can log
|
||||||
|
and count it separately from a genuinely worded diagram.
|
||||||
|
"""
|
||||||
|
|
||||||
|
text: str
|
||||||
|
structural: bool
|
||||||
|
|
||||||
|
|
||||||
|
def local_name(tag: object) -> str:
|
||||||
|
"""The namespace-agnostic name of an element tag.
|
||||||
|
|
||||||
|
``{http://www.w3.org/2000/svg}title`` and ``title`` both give
|
||||||
|
``"title"`` — ElementTree expands every declared prefix (including
|
||||||
|
``<svg:title>``) into Clark notation, so stripping the ``{uri}`` is
|
||||||
|
all "namespace-agnostic" needs. A non-string tag (comment /
|
||||||
|
processing-instruction marker) has no name → ``""``.
|
||||||
|
"""
|
||||||
|
if not isinstance(tag, str):
|
||||||
|
return ""
|
||||||
|
return tag.rsplit("}", 1)[-1]
|
||||||
|
|
||||||
|
|
||||||
|
def extract_svg_text(data: bytes, *, stem: str = "") -> SvgText | None:
|
||||||
|
"""Turn raw SVG bytes into the document's searchable text.
|
||||||
|
|
||||||
|
Returns ``None`` when the input is unusable or unsafe: empty /
|
||||||
|
whitespace-only, larger than :data:`MAX_SVG_BYTES`, carrying a
|
||||||
|
``<!DOCTYPE``/``<!ENTITY`` declaration, or not well-formed XML. The
|
||||||
|
caller's fail-soft (skip the document, count it) is the honest
|
||||||
|
outcome — this function never raises for bad input.
|
||||||
|
|
||||||
|
*stem* is the file name without extension: it names the document in
|
||||||
|
the structural fallback line and is never invented.
|
||||||
|
|
||||||
|
Otherwise a :class:`SvgText` whose ``text`` is the document-order
|
||||||
|
``<title>``/``<desc>``/``<text>`` words (``structural=False``), or —
|
||||||
|
when the file carries no words at all — the structural line built
|
||||||
|
from the stem, the shape census and the ``viewBox``
|
||||||
|
(``structural=True``, LOCKED A2).
|
||||||
|
"""
|
||||||
|
if not data or not data.strip():
|
||||||
|
return None
|
||||||
|
if len(data) > MAX_SVG_BYTES:
|
||||||
|
return None
|
||||||
|
lowered = data.lower()
|
||||||
|
if _DOCTYPE_MARKER in lowered or _ENTITY_MARKER in lowered:
|
||||||
|
return None
|
||||||
|
try:
|
||||||
|
root = ElementTree.fromstring(data)
|
||||||
|
except ElementTree.ParseError:
|
||||||
|
return None
|
||||||
|
items = _dedupe(_walk_text(root))
|
||||||
|
if items:
|
||||||
|
return SvgText(text=_cap_text("\n".join(items)), structural=False)
|
||||||
|
return SvgText(text=_structural_line(root, stem), structural=True)
|
||||||
|
|
||||||
|
|
||||||
|
def _walk_text(root: ElementTree.Element) -> list[str]:
|
||||||
|
"""Every worded ``title``/``desc``/``text`` block, in document order.
|
||||||
|
|
||||||
|
Items are whitespace-normalised; a block that normalises to nothing
|
||||||
|
(an empty ``<title></title>``) is dropped so the fallback can still
|
||||||
|
describe the file.
|
||||||
|
"""
|
||||||
|
items: list[str] = []
|
||||||
|
for el in root.iter():
|
||||||
|
name = local_name(el.tag)
|
||||||
|
if name in _BLOCK_TEXT_NAMES:
|
||||||
|
raw = "".join(el.itertext())
|
||||||
|
elif name in _SPACED_TEXT_NAMES:
|
||||||
|
raw = " ".join(el.itertext())
|
||||||
|
else:
|
||||||
|
continue
|
||||||
|
normalised = " ".join(raw.split())
|
||||||
|
if normalised:
|
||||||
|
items.append(normalised)
|
||||||
|
return items
|
||||||
|
|
||||||
|
|
||||||
|
def _dedupe(items: list[str]) -> list[str]:
|
||||||
|
"""Drop repeats case-insensitively, keeping the FIRST occurrence.
|
||||||
|
|
||||||
|
A ``<title>`` mirrored in a ``<desc>`` (exporters do this) is one
|
||||||
|
sentence in the index, not two; the first position in document
|
||||||
|
order is the one kept.
|
||||||
|
"""
|
||||||
|
seen: set[str] = set()
|
||||||
|
unique: list[str] = []
|
||||||
|
for item in items:
|
||||||
|
key = item.casefold()
|
||||||
|
if key in seen:
|
||||||
|
continue
|
||||||
|
seen.add(key)
|
||||||
|
unique.append(item)
|
||||||
|
return unique
|
||||||
|
|
||||||
|
|
||||||
|
def _cap_text(text: str) -> str:
|
||||||
|
"""Cap *text* at :data:`MAX_SVG_TEXT_CHARS` without cutting a word.
|
||||||
|
|
||||||
|
An over-cap text is cut at the LAST newline or period before the cap
|
||||||
|
(whichever comes later) so the stored text ends on a boundary; only
|
||||||
|
a window with no boundary at all (one colossal run of characters)
|
||||||
|
is hard-cut.
|
||||||
|
"""
|
||||||
|
if len(text) <= MAX_SVG_TEXT_CHARS:
|
||||||
|
return text
|
||||||
|
window = text[:MAX_SVG_TEXT_CHARS]
|
||||||
|
cut = max(window.rfind("\n"), window.rfind("."))
|
||||||
|
if cut < 0:
|
||||||
|
return window
|
||||||
|
return text[: cut + 1].rstrip()
|
||||||
|
|
||||||
|
|
||||||
|
def _structural_line(root: ElementTree.Element, stem: str) -> str:
|
||||||
|
"""The LOCKED A2 fallback: one line of facts, nothing invented.
|
||||||
|
|
||||||
|
``Vector diagram "<stem>" — <N> shapes (<top-3 kinds>), viewBox
|
||||||
|
<w>×<h>`` — each fact clause is omitted when the file does not
|
||||||
|
state it, and the stem clause disappears entirely when empty (no
|
||||||
|
empty quotes). With no facts and no stem the line is just
|
||||||
|
``Vector diagram``.
|
||||||
|
"""
|
||||||
|
head = f'Vector diagram "{stem}"' if stem else "Vector diagram"
|
||||||
|
facts = [c for c in (_shape_clause(root), _viewbox_clause(root)) if c]
|
||||||
|
if not facts:
|
||||||
|
return head
|
||||||
|
return f"{head} — {', '.join(facts)}"
|
||||||
|
|
||||||
|
|
||||||
|
def _shape_clause(root: ElementTree.Element) -> str:
|
||||||
|
"""``<N> shapes (<kinds>)`` — the drawing's element census.
|
||||||
|
|
||||||
|
*N* counts every descendant element except ``svg``, ``title`` and
|
||||||
|
``desc`` (structure, not drawing). The kinds are the
|
||||||
|
:data:`_TOP_SHAPE_NAMES` most common local names, ordered by count
|
||||||
|
(document order breaks ties) and annotated with their count — the
|
||||||
|
count is left off a kind that appears exactly once, which is what
|
||||||
|
the total already says. ``""`` when there are no shapes at all, so
|
||||||
|
the clause is omitted rather than claiming "0 shapes".
|
||||||
|
"""
|
||||||
|
counts: dict[str, int] = {}
|
||||||
|
first_seen: dict[str, int] = {}
|
||||||
|
total = 0
|
||||||
|
for index, el in enumerate(root.iter()):
|
||||||
|
if el is root:
|
||||||
|
continue
|
||||||
|
name = local_name(el.tag)
|
||||||
|
if not name or name in _NON_SHAPE_NAMES:
|
||||||
|
continue
|
||||||
|
counts[name] = counts.get(name, 0) + 1
|
||||||
|
first_seen.setdefault(name, index)
|
||||||
|
total += 1
|
||||||
|
if not total:
|
||||||
|
return ""
|
||||||
|
top = sorted(counts, key=lambda name: (-counts[name], first_seen[name]))
|
||||||
|
listing = ", ".join(
|
||||||
|
f"{name}{_TIMES}{counts[name]}" if counts[name] > 1 else name
|
||||||
|
for name in top[:_TOP_SHAPE_NAMES]
|
||||||
|
)
|
||||||
|
noun = "shape" if total == 1 else "shapes"
|
||||||
|
return f"{total} {noun} ({listing})"
|
||||||
|
|
||||||
|
|
||||||
|
def _viewbox_clause(root: ElementTree.Element) -> str:
|
||||||
|
"""``viewBox <w>×<h>`` from the root's ``viewBox``, or ``""``.
|
||||||
|
|
||||||
|
The spec grammar is four whitespace-and/or-comma separated numbers;
|
||||||
|
the last two are the width and height. Unparseable (wrong count,
|
||||||
|
non-numeric, non-finite) → ``""`` — the clause is omitted rather
|
||||||
|
than reporting a size the file does not state. Integral values
|
||||||
|
print without a trailing ``.0``.
|
||||||
|
"""
|
||||||
|
raw = (root.get("viewBox") or "").strip()
|
||||||
|
if not raw:
|
||||||
|
return ""
|
||||||
|
parts = [part for part in _VIEWBOX_SPLIT_RE.split(raw) if part]
|
||||||
|
if len(parts) != 4:
|
||||||
|
return ""
|
||||||
|
try:
|
||||||
|
_min_x, _min_y, width, height = (float(part) for part in parts)
|
||||||
|
except ValueError:
|
||||||
|
return ""
|
||||||
|
if not (math.isfinite(width) and math.isfinite(height)):
|
||||||
|
return ""
|
||||||
|
return f"viewBox {_format_number(width)}{_TIMES}{_format_number(height)}"
|
||||||
|
|
||||||
|
|
||||||
|
def _format_number(value: float) -> str:
|
||||||
|
"""``8`` for 8.0, ``8.5`` for 8.5 — no trailing ``.0`` in a size."""
|
||||||
|
return str(int(value)) if value.is_integer() else str(value)
|
||||||
+11
-1
@@ -2498,7 +2498,17 @@ function appendRelated(wrap, related) {
|
|||||||
* replaced, in place, with a plain `.source-chip` anchor for the same
|
* replaced, in place, with a plain `.source-chip` anchor for the same
|
||||||
* document — the citation survives a lost byte copy, never a
|
* document — the citation survives a lost byte copy, never a
|
||||||
* broken-image icon (the row's bytes route 404s when the copy was
|
* broken-image icon (the row's bytes route 404s when the copy was
|
||||||
* lost). */
|
* lost).
|
||||||
|
*
|
||||||
|
* Phase 181 (task 04, design D7): DELIBERATE NON-CHANGE for SVG. The
|
||||||
|
* figure keys off image_url alone, and a size-less (viewBox-only)
|
||||||
|
* vector drawing scales INSIDE the fixed cap of .source-image-img
|
||||||
|
* (max-height: 96px + max-width: 100% + object-fit: contain), so the
|
||||||
|
* compact chat citation needs no format-aware sizing; the full-size
|
||||||
|
* preview is the modal this figure/chip opens (doc.format is known
|
||||||
|
* there — see docImageBlock in document.js). NO format field is
|
||||||
|
* added to SourceRef — the wire contract stays byte-identical for
|
||||||
|
* every text doc (the phase-83/122 contract). */
|
||||||
function appendSourceImageFigure(meta, s) {
|
function appendSourceImageFigure(meta, s) {
|
||||||
const label = `${s.source}/${s.path}`;
|
const label = `${s.source}/${s.path}`;
|
||||||
const fig = document.createElement("a");
|
const fig = document.createElement("a");
|
||||||
|
|||||||
@@ -573,7 +573,26 @@ function docImageBlock(doc) {
|
|||||||
const wrap = document.createElement("div");
|
const wrap = document.createElement("div");
|
||||||
wrap.className = "doc-image";
|
wrap.className = "doc-image";
|
||||||
const img = document.createElement("img");
|
const img = document.createElement("img");
|
||||||
img.className = "doc-image-img";
|
// Phase 181 (task 04, design D6/D7): SVG is a VECTOR image doc. A
|
||||||
|
// drawing carrying only a viewBox has NO intrinsic size, so the
|
||||||
|
// default replaced-element box (300×150) would sit small inside the
|
||||||
|
// full-width .doc-image card; the --vector modifier (the base class
|
||||||
|
// is KEPT so every current rule still applies) lets width: 100%
|
||||||
|
// span the card — the full-size preview. doc.format is the ONLY
|
||||||
|
// place in the app that knows the type: the bytes URL is
|
||||||
|
// extension-less and SourceRef carries no format field by design
|
||||||
|
// (D7). LOCKED A3: the SVG is NEVER inlined into the DOM —
|
||||||
|
// img.src = doc.image_url stays createElement + property
|
||||||
|
// assignment (this file's XSS contract, unchanged). Legacy
|
||||||
|
// honesty (D6): a row still indexed as TEXT (an .svg under
|
||||||
|
// BOR_IMAGES=0, or not yet re-synced) keeps rendering its XML in
|
||||||
|
// .doc-raw — the truth about that row (there are no bytes behind
|
||||||
|
// it); task 02's kind-mismatch guard converts it on the next sync.
|
||||||
|
// Nobody "fixes" that by sniffing the content string.
|
||||||
|
img.className =
|
||||||
|
doc.format === "svg"
|
||||||
|
? "doc-image-img doc-image-img--vector"
|
||||||
|
: "doc-image-img";
|
||||||
img.src = doc.image_url;
|
img.src = doc.image_url;
|
||||||
img.alt =
|
img.alt =
|
||||||
typeof doc.summary === "string" && doc.summary.trim() !== ""
|
typeof doc.summary === "string" && doc.summary.trim() !== ""
|
||||||
|
|||||||
@@ -2324,7 +2324,12 @@ export async function mount(root) {
|
|||||||
// Phase 122 (task 04): an image row gets the FIXED 48px thumbnail
|
// Phase 122 (task 04): an image row gets the FIXED 48px thumbnail
|
||||||
// box before the path link (the .kb-doc-path flex wrapper — the
|
// box before the path link (the .kb-doc-path flex wrapper — the
|
||||||
// link keeps its ellipsis). Text rows keep the bare-link cell,
|
// link keeps its ellipsis). Text rows keep the bare-link cell,
|
||||||
// byte-identical to pre-phase (no box at all).
|
// byte-identical to pre-phase (no box at all). Phase 181 (task
|
||||||
|
// 04, design D7): verified NON-CHANGE for SVG rows — the thumb
|
||||||
|
// loads the same extension-less image_url and .kb-doc-thumb-img
|
||||||
|
// pins the img to the fixed 48px box (width/height 100% +
|
||||||
|
// object-fit: cover), so a size-less viewBox-only vector fills
|
||||||
|
// the box like a raster; no format sniffing needed here either.
|
||||||
if (d.is_image) {
|
if (d.is_image) {
|
||||||
const pathWrap = document.createElement("div");
|
const pathWrap = document.createElement("div");
|
||||||
pathWrap.className = "kb-doc-path";
|
pathWrap.className = "kb-doc-path";
|
||||||
|
|||||||
@@ -5742,6 +5742,23 @@ a.answer-image:hover { text-decoration: underline; } /* flat — the caption und
|
|||||||
border-radius: var(--radius-sm);
|
border-radius: var(--radius-sm);
|
||||||
background: var(--bg);
|
background: var(--bg);
|
||||||
}
|
}
|
||||||
|
/* The vector variant (phase 181, task 04, design D6 — applied in
|
||||||
|
document.js when doc.format === "svg", base class kept). An SVG
|
||||||
|
carrying only a viewBox has NO intrinsic size, so without an
|
||||||
|
explicit width the default replaced-element box (300×150) applies
|
||||||
|
and the drawing sits small inside the full-width .doc-image card;
|
||||||
|
width: 100% spans the card (the full-size preview the owner
|
||||||
|
asked for). object-fit: contain protects the drawing from being
|
||||||
|
squashed when the base rule's max-height: 70vh clamps a tall
|
||||||
|
vector — the box letterboxes, the graphic keeps its aspect ratio.
|
||||||
|
Static content — no animation (nothing for prefers-reduced-motion
|
||||||
|
to still). Palette-neutral: no colour here — the base rule's --bg
|
||||||
|
shows through (phase-08 tokens only, no new hue). */
|
||||||
|
.doc-image-img--vector {
|
||||||
|
width: 100%;
|
||||||
|
height: auto;
|
||||||
|
object-fit: contain;
|
||||||
|
}
|
||||||
/* The onerror fallback (the route's 404 — the row exists but the
|
/* The onerror fallback (the route's 404 — the row exists but the
|
||||||
copy was lost): a muted note in the card's place-of-image; the
|
copy was lost): a muted note in the card's place-of-image; the
|
||||||
description below still renders, so the doc stays readable. Phase
|
description below still renders, so the doc stays readable. Phase
|
||||||
|
|||||||
@@ -0,0 +1,607 @@
|
|||||||
|
"""Phase 181 E2E (Playwright): SVG documents preview as graphics, not XML.
|
||||||
|
|
||||||
|
Source: ``TODO.md`` L3 — "Need SVG render support on the front end when
|
||||||
|
citing and svg source. Also would be nice to click an svg and see a
|
||||||
|
preview in the RAG tab rather than the xml".
|
||||||
|
Run in isolation (DB must be up: ``podman compose up -d db``):
|
||||||
|
|
||||||
|
uv run pytest tests/e2e/test_svg_document_preview.py -v --no-cov
|
||||||
|
|
||||||
|
The suite's OWN module app (the phase-122 ``test_image_documents.py``
|
||||||
|
pattern) runs with ``BOR_IMAGES=true`` and every suite-writable home
|
||||||
|
(``BOR_IMAGE_DIR`` included) pinned into the suite's tmp dir — process
|
||||||
|
env ranks above an operator's local gitignored ``.env``, so the toggle
|
||||||
|
cannot leak in either direction. Seeding is the phase-138 pattern
|
||||||
|
(``test_markdown_blockquotes`` / ``test_markdown_inline_links_images``):
|
||||||
|
truncate, then run the REAL importer in-process against a COPY of the
|
||||||
|
phase's dedicated fixture tree ``tests/fixtures/svg_docs/`` (the shared
|
||||||
|
fixture set's counts are pinned by other suites and must not move) with
|
||||||
|
the deterministic mock embeddings — the vector branch of task 02
|
||||||
|
indexes both SVGs with NO vision call, the PNG half of the last test is
|
||||||
|
described by the mock's ``IMAGE_DESCRIPTION`` branch.
|
||||||
|
|
||||||
|
Test → story mapping (Playwright Mapping Rule):
|
||||||
|
1. ``test_viewer_renders_svg_not_xml`` — the Sources row for
|
||||||
|
``diagrams/topology.svg`` opens the modal showing
|
||||||
|
``img.doc-image-img--vector`` (``naturalWidth > 0``, non-empty alt),
|
||||||
|
NO ``pre.doc-raw`` and no ``<svg`` XML text anywhere in the modal
|
||||||
|
body, the extracted ``<title>`` text below the graphic; the "Full
|
||||||
|
page" escape hatch renders the same graphic at ≥ 600px width at a
|
||||||
|
1280px viewport (the task-04 vector sizing contract — a
|
||||||
|
``viewBox``-only SVG would otherwise sit in the 300×150 default box).
|
||||||
|
2. ``test_chat_figure_and_modal_preview`` — the grounded scripted-read
|
||||||
|
question cites the SVG doc as the ``.source-image`` figure (the
|
||||||
|
``<img>`` loads, the caption is the extracted text), and clicking
|
||||||
|
the figure opens the modal with the full graphic (the "preview in
|
||||||
|
the RAG tab" contract); Escape closes it.
|
||||||
|
3. ``test_blank_svg_is_indexed_and_searchable`` — ``diagrams/blank.svg``
|
||||||
|
(pure paths) exists as an image document whose content is the
|
||||||
|
structural LOCKED A2 line ``Vector diagram "blank" — …``, is listed
|
||||||
|
on the Sources page with its thumbnail, and opens in the modal with
|
||||||
|
that line as its description.
|
||||||
|
4. ``test_markdown_image_ref_renders`` — the two-stage seeding: import
|
||||||
|
the tree, read the SVG doc's id from the DB, write
|
||||||
|
``notes/inline.md`` containing ````
|
||||||
|
and re-import; the viewer for that document shows ``img.md-img``
|
||||||
|
pointing at the bytes route and LOADING (``naturalWidth > 0``) — the
|
||||||
|
phase-180 renderer + this route end to end.
|
||||||
|
5. ``test_raster_surfaces_unchanged`` — the smoke guard: a PNG doc in
|
||||||
|
the same tree still renders through ``.doc-image-img`` (no
|
||||||
|
``--vector`` class) and its bytes route carries NO sandbox CSP (the
|
||||||
|
app-wide policy only) — phase 122/124 behaviour byte-unchanged.
|
||||||
|
"""
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import asyncio
|
||||||
|
import base64
|
||||||
|
import json
|
||||||
|
import os
|
||||||
|
import shutil
|
||||||
|
import subprocess
|
||||||
|
import sys
|
||||||
|
from collections.abc import Iterator
|
||||||
|
from pathlib import Path
|
||||||
|
from threading import Thread
|
||||||
|
from typing import Any
|
||||||
|
|
||||||
|
import httpx
|
||||||
|
import pytest
|
||||||
|
from playwright.sync_api import Page, expect
|
||||||
|
from sqlalchemy import select, text
|
||||||
|
|
||||||
|
from app.config import Settings
|
||||||
|
from app.db import SessionLocal
|
||||||
|
from app.models import Chunk, Document
|
||||||
|
from app.rag.agent import IMAGE_DOC_MARKER
|
||||||
|
from app.rag.importer import ImportSummary, import_sources
|
||||||
|
from app.rag.llm import LLMClient
|
||||||
|
from app.rag.svg_text import extract_svg_text
|
||||||
|
from e2e.auth_helpers import login
|
||||||
|
from e2e.conftest import ADMIN_PASSWORD, SESSION_SECRET, USE_REAL_LLM, _wait_http
|
||||||
|
|
||||||
|
if USE_REAL_LLM:
|
||||||
|
pytest.skip(
|
||||||
|
"phase 181 pins the deterministic mock's grounded turn and seeded "
|
||||||
|
"figures — the real model answers differently",
|
||||||
|
allow_module_level=True,
|
||||||
|
)
|
||||||
|
|
||||||
|
REPO = Path(__file__).resolve().parents[2]
|
||||||
|
FIXTURES = REPO / "tests" / "fixtures" / "svg_docs"
|
||||||
|
|
||||||
|
#: The source name = the imported tree root's basename (the importer's
|
||||||
|
#: ``archive_source_name``/directory rule — the phase-180 pattern).
|
||||||
|
SOURCE = "svg_docs"
|
||||||
|
TOPO_PATH = "diagrams/topology.svg"
|
||||||
|
BLANK_PATH = "diagrams/blank.svg"
|
||||||
|
|
||||||
|
#: The fixture diagram's own words — what the vector branch indexes
|
||||||
|
#: (LOCKED A1) and what every text assertion below pins. Pinned as a
|
||||||
|
#: literal here (not re-derived from the extractor) so a change to the
|
||||||
|
#: extraction contract breaks this suite LOUDLY, next to the product.
|
||||||
|
TOPO_TEXT = (
|
||||||
|
"Homelab network topology\n"
|
||||||
|
"Wiring diagram of the homelab: the edge router uplinks two managed "
|
||||||
|
"switches, the hypervisor rack and the nas storage shelf.\n"
|
||||||
|
"rack-1 switch\n"
|
||||||
|
"nas storage"
|
||||||
|
)
|
||||||
|
|
||||||
|
#: The grounded scripted-read question (phase-118 A1: a zero-read turn
|
||||||
|
#: chips nothing — the mock emits the scripted ``read`` then echoes the
|
||||||
|
#: result, the phase-122 figure precedent). Token-overlap cosines
|
||||||
|
#: against the mock embeddings: topology.svg ≈0.70, the infra.md text
|
||||||
|
#: co-citation ≈0.72 (both over the mock-calibrated 0.30 gate); the
|
||||||
|
#: blank.svg line sits at ≈0.04 and never pollutes the citation set.
|
||||||
|
QUESTION = (
|
||||||
|
"Read the suggested document: read svg_docs/diagrams/topology.svg — "
|
||||||
|
"what does the homelab network topology diagram show about the edge "
|
||||||
|
"router, rack-1 switch and the nas storage?"
|
||||||
|
)
|
||||||
|
|
||||||
|
#: A real 1×1 PNG (the phase-122/124 suites' bytes) — the raster smoke
|
||||||
|
#: guard of the last test, written into the suite's tree copy only there.
|
||||||
|
PNG_1X1 = base64.b64decode(
|
||||||
|
"iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAYAAAAfFcSJ"
|
||||||
|
"AAAAC0lEQVR4nGP4DwQACfsD/fteaysAAAAASUVORK5CYII="
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# Module app: BOR_IMAGES pinned true, the image dir in the suite's scratch
|
||||||
|
# (the in-process importer and the app under test SHARE that directory —
|
||||||
|
# the copies the seed writes are the bytes the route serves).
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
def _app_env(mock_llm: int, port: int, scratch: Path) -> dict[str, str]:
|
||||||
|
"""The conftest app env (leak guards included) with the phase-181
|
||||||
|
knobs: ``BOR_IMAGES=true`` EXPLICITLY (an operator's local ``.env``
|
||||||
|
cannot leak the toggle off) and ``BOR_IMAGE_DIR`` pinned into the
|
||||||
|
suite's scratch (the app must never read or write the owner's real
|
||||||
|
image home)."""
|
||||||
|
env = dict(os.environ)
|
||||||
|
env.pop("DEBUGPY", None)
|
||||||
|
env["BOR_ENVIRONMENT"] = "e2e"
|
||||||
|
env["BOR_STATIC_DIR"] = str(REPO / "frontend")
|
||||||
|
env["BOR_LLM_BASE_URL"] = f"http://127.0.0.1:{mock_llm}/v1"
|
||||||
|
# Mock-calibrated gate (conftest pattern) — QUESTION grounds on these.
|
||||||
|
env["BOR_RELEVANCE_THRESHOLD"] = "0.30"
|
||||||
|
env["BOR_LEXICAL_SUPPORT_FLOOR"] = "0.15"
|
||||||
|
env["BOR_SOURCE_USEFULNESS_FLOOR"] = "0.15"
|
||||||
|
env["BOR_LLM_RETRY_DELAY"] = "0"
|
||||||
|
env["BOR_LLM_RETRIES"] = str(Settings.model_fields["llm_retries"].default)
|
||||||
|
env.setdefault(
|
||||||
|
"BOR_DATABASE_URL",
|
||||||
|
"postgresql+psycopg://reese:reese@localhost:5432/brain_of_reese",
|
||||||
|
)
|
||||||
|
env["BOR_ADMIN_PASSWORD"] = ADMIN_PASSWORD
|
||||||
|
env["BOR_SESSION_SECRET"] = SESSION_SECRET
|
||||||
|
# Leak guards (conftest pattern).
|
||||||
|
env["BOR_GIT_SOURCES"] = ""
|
||||||
|
env["BOR_DOCS_REPO"] = ""
|
||||||
|
env["BOR_SUGGESTIONS"] = json.dumps(
|
||||||
|
Settings.model_fields["suggestions"].default
|
||||||
|
)
|
||||||
|
env["BOR_INPUT_PLACEHOLDER"] = Settings.model_fields["input_placeholder"].default
|
||||||
|
env["BOR_FOOTER_TEXT"] = Settings.model_fields["footer_text"].default
|
||||||
|
# Phase 181: the toggle under test + the suite-private homes.
|
||||||
|
env["BOR_IMAGES"] = "true"
|
||||||
|
env["BOR_UPLOAD_DIR"] = str(scratch / "uploads")
|
||||||
|
env["BOR_SOURCES_DIR"] = str(scratch / "checkouts")
|
||||||
|
env["BOR_IMAGE_DIR"] = str(_image_dir(scratch))
|
||||||
|
return env
|
||||||
|
|
||||||
|
|
||||||
|
def _image_dir(scratch: Path) -> Path:
|
||||||
|
return scratch / "images"
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.fixture(scope="module")
|
||||||
|
def scratch(tmp_path_factory: pytest.TempPathFactory) -> Path:
|
||||||
|
return tmp_path_factory.mktemp("bor_svg_preview")
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.fixture(scope="module")
|
||||||
|
def app_server(
|
||||||
|
mock_llm: int, scratch: Path
|
||||||
|
) -> Iterator[str]:
|
||||||
|
"""The suite's own app — ``BOR_IMAGES=true`` (the conftest session app
|
||||||
|
is never started in this isolated run, so no port clash)."""
|
||||||
|
port = int(os.environ.get("E2E_APP_PORT_SVG_PREVIEW", "8196"))
|
||||||
|
proc = subprocess.Popen(
|
||||||
|
[sys.executable, "-m", "uvicorn", "app.main:app",
|
||||||
|
"--host", "127.0.0.1", "--port", str(port), "--log-level", "warning"],
|
||||||
|
cwd=REPO,
|
||||||
|
env=_app_env(mock_llm, port, scratch),
|
||||||
|
)
|
||||||
|
try:
|
||||||
|
_wait_http(f"http://127.0.0.1:{port}/api/health")
|
||||||
|
yield f"http://127.0.0.1:{port}"
|
||||||
|
finally:
|
||||||
|
proc.terminate()
|
||||||
|
try:
|
||||||
|
proc.wait(timeout=10)
|
||||||
|
except subprocess.TimeoutExpired:
|
||||||
|
proc.kill()
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.fixture(scope="module")
|
||||||
|
def app_url(app_server: str) -> str:
|
||||||
|
return app_server
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# Seeding: the phase-138 truncate-then-import pattern against a per-test
|
||||||
|
# COPY of the dedicated fixture tree (the two-stage test appends a file to
|
||||||
|
# the copy, never to the tracked fixture).
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
def _fresh_tree(scratch: Path) -> Path:
|
||||||
|
"""A pristine copy of the fixture tree at ``<scratch>/src/svg_docs``
|
||||||
|
(the basename IS the source name). Rmtree-first: the two-stage test
|
||||||
|
leaves an ``inline.md`` behind, and every other test must not see it."""
|
||||||
|
tree = scratch / "src" / SOURCE
|
||||||
|
if tree.exists():
|
||||||
|
shutil.rmtree(tree)
|
||||||
|
tree.parent.mkdir(parents=True, exist_ok=True)
|
||||||
|
shutil.copytree(FIXTURES, tree)
|
||||||
|
return tree
|
||||||
|
|
||||||
|
|
||||||
|
async def _import(tree: Path, mock_llm: int, scratch: Path) -> ImportSummary:
|
||||||
|
"""The REAL importer over the fixture copy (task 02's vector branch
|
||||||
|
runs in-process against the mock — the same LLM the app under test
|
||||||
|
talks to). A coroutine: the caller runs it through
|
||||||
|
:func:`_run_in_thread` (Playwright owns the test thread's loop)."""
|
||||||
|
settings = Settings(
|
||||||
|
# the owner's .env cannot leak the vector knobs (_env_file is a
|
||||||
|
# runtime-only pydantic-settings kwarg — pyright cannot see it).
|
||||||
|
_env_file=None, # pyright: ignore[reportCallIssue]
|
||||||
|
llm_base_url=f"http://127.0.0.1:{mock_llm}/v1",
|
||||||
|
images=True, # the toggle under test (LOCKED A7's on state)
|
||||||
|
image_dir=str(_image_dir(scratch)),
|
||||||
|
image_extensions=Settings.model_fields["image_extensions"].default,
|
||||||
|
)
|
||||||
|
return await import_sources([tree], LLMClient(settings))
|
||||||
|
|
||||||
|
|
||||||
|
def _run_in_thread(coro: Any) -> Any:
|
||||||
|
"""Run a coroutine on a worker thread — Playwright's sync API keeps an
|
||||||
|
asyncio loop on the test thread (the house helper, phase 180)."""
|
||||||
|
box: dict[str, Any] = {}
|
||||||
|
|
||||||
|
def runner() -> None:
|
||||||
|
try:
|
||||||
|
box["value"] = asyncio.run(coro)
|
||||||
|
except BaseException as e: # noqa: BLE001 — re-raised on the test thread
|
||||||
|
box["error"] = e
|
||||||
|
|
||||||
|
t = Thread(target=runner)
|
||||||
|
t.start()
|
||||||
|
t.join()
|
||||||
|
if "error" in box:
|
||||||
|
raise box["error"]
|
||||||
|
return box["value"]
|
||||||
|
|
||||||
|
|
||||||
|
def _truncate() -> None:
|
||||||
|
"""Fresh KB per test (the suites share one Postgres). ``sources_meta``
|
||||||
|
resets with the KB — the re-import's bump UPSERTS the row back
|
||||||
|
(the phase-122 truncate list, ``bump_sources_version``)."""
|
||||||
|
with SessionLocal() as db:
|
||||||
|
db.execute(
|
||||||
|
text(
|
||||||
|
"TRUNCATE chunks, documents, query_log, steering_notes, "
|
||||||
|
"kb_overview, folder_summaries, sources_meta"
|
||||||
|
)
|
||||||
|
)
|
||||||
|
db.commit()
|
||||||
|
|
||||||
|
|
||||||
|
def _seed(mock_llm: int, scratch: Path) -> ImportSummary:
|
||||||
|
"""Truncate + import the fresh fixture copy. Both SVGs arrive through
|
||||||
|
the VECTOR branch (``svg_indexed == 2`` — no vision call happened),
|
||||||
|
infra.md as the text co-citation: three docs total."""
|
||||||
|
_truncate()
|
||||||
|
summary = _run_in_thread(_import(_fresh_tree(scratch), mock_llm, scratch))
|
||||||
|
assert (summary.files, summary.added, summary.images_failed) == (3, 3, 0)
|
||||||
|
assert summary.svg_indexed == 2, summary
|
||||||
|
return summary
|
||||||
|
|
||||||
|
|
||||||
|
def _doc(source: str, path: str) -> Document:
|
||||||
|
with SessionLocal() as db:
|
||||||
|
doc = db.scalar(
|
||||||
|
select(Document).where(Document.source == source, Document.path == path)
|
||||||
|
)
|
||||||
|
assert doc is not None, f"no documents row for ({source!r}, {path!r})"
|
||||||
|
return doc
|
||||||
|
|
||||||
|
|
||||||
|
def _health_db(app_url: str) -> bool:
|
||||||
|
try:
|
||||||
|
return httpx.get(f"{app_url}/api/health", timeout=5).json()["db"] == "up"
|
||||||
|
except Exception: # noqa: BLE001 — unreachable is the skip case
|
||||||
|
return False
|
||||||
|
|
||||||
|
|
||||||
|
def _skip_without_db(app_url: str) -> None:
|
||||||
|
if not _health_db(app_url):
|
||||||
|
pytest.skip("Postgres not reachable — run `podman compose up -d db` first")
|
||||||
|
|
||||||
|
|
||||||
|
def _drill(page: Page, name: str) -> None:
|
||||||
|
"""One level of the KB tree (the phase-122/180 click, made settle-
|
||||||
|
safe): the drill keeps rendering the CURRENT level while the node
|
||||||
|
fetch runs (phase 167), so the level is settled exactly when the
|
||||||
|
clicked link is gone from the replaced table (intermediate levels
|
||||||
|
with no direct files must not wait on an empty ``#docs-tbody``)."""
|
||||||
|
link = page.locator(f'#folders-tbody a.folder-link:text-is("{name}")')
|
||||||
|
expect(link).to_be_visible(timeout=30_000)
|
||||||
|
link.click()
|
||||||
|
expect(link).to_have_count(0)
|
||||||
|
|
||||||
|
|
||||||
|
def _open_doc_modal(page: Page, app_url: str, folder: str, doc_path: str) -> Any:
|
||||||
|
"""Sources → source → folder → the document's row → the same-page
|
||||||
|
modal. Returns the modal locator (loaded)."""
|
||||||
|
login(page, app_url) # lands on /sources.html (admin)
|
||||||
|
_drill(page, SOURCE)
|
||||||
|
_drill(page, folder)
|
||||||
|
row = page.locator("#docs-tbody tr", has_text=doc_path)
|
||||||
|
expect(row).to_have_count(1)
|
||||||
|
row.locator("td:nth-child(2) a.doc-link").click()
|
||||||
|
modal = page.locator("#doc-modal")
|
||||||
|
expect(modal).to_be_visible(timeout=30_000)
|
||||||
|
return modal
|
||||||
|
|
||||||
|
|
||||||
|
def _seed_page(page: Page, app_url: str, mock_llm: int, scratch: Path) -> None:
|
||||||
|
_skip_without_db(app_url)
|
||||||
|
page.set_default_timeout(30_000)
|
||||||
|
_seed(mock_llm, scratch)
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# 1. The viewer: the SVG paints as a graphic — the XML is nowhere
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
def test_viewer_renders_svg_not_xml(
|
||||||
|
page: Page, app_url: str, mock_llm: int, scratch: Path
|
||||||
|
) -> None:
|
||||||
|
"""TODO.md L3's viewer half: the modal shows the RENDERED vector
|
||||||
|
graphic (the ``--vector`` sizing class, task 04) from the bytes
|
||||||
|
route, alt = the extracted text; NO raw-XML surface exists — no
|
||||||
|
``pre.doc-raw``, no ``<svg`` markup text anywhere in the modal body
|
||||||
|
— and the extracted ``<title>`` sentence reads as the description
|
||||||
|
below the graphic. The "Full page" escape hatch (the
|
||||||
|
``/document.html`` viewer) spans the drawing: ≥ 600px wide at a
|
||||||
|
1280px viewport, where the 300×150 replaced-element default box
|
||||||
|
would have sat small (the phase's sizing contract)."""
|
||||||
|
_seed_page(page, app_url, mock_llm, scratch)
|
||||||
|
doc = _doc(SOURCE, TOPO_PATH)
|
||||||
|
assert doc.is_image is True # the premise: a vector IMAGE row
|
||||||
|
|
||||||
|
modal = _open_doc_modal(page, app_url, "diagrams", TOPO_PATH)
|
||||||
|
expect(page.locator("#doc-modal-title")).to_have_text("topology")
|
||||||
|
|
||||||
|
img = modal.locator("img.doc-image-img--vector")
|
||||||
|
expect(img).to_have_count(1)
|
||||||
|
expect(img).to_be_visible(timeout=15_000)
|
||||||
|
assert img.evaluate("el => el.naturalWidth") > 0, "the SVG bytes did not paint"
|
||||||
|
assert img.evaluate("el => el.naturalHeight") > 0
|
||||||
|
alt = img.get_attribute("alt")
|
||||||
|
assert alt is not None and "Homelab network topology" in alt
|
||||||
|
assert img.get_attribute("src") == f"/api/documents/{doc.id}/image"
|
||||||
|
expect(modal.locator(".doc-image-unavailable")).to_have_count(0)
|
||||||
|
|
||||||
|
# The honest negative (the owner's complaint, verbatim): NO XML text
|
||||||
|
# and no raw-code surface for an SVG row.
|
||||||
|
expect(modal.locator("pre.doc-raw")).to_have_count(0)
|
||||||
|
body_text = modal.inner_text()
|
||||||
|
assert "<svg" not in body_text, "raw XML text painted into the modal"
|
||||||
|
assert "</title>" not in body_text
|
||||||
|
# The extracted words ARE the document's readable content, below
|
||||||
|
# the graphic (the phase-165 prose arm — summary === content).
|
||||||
|
prose = modal.locator(".doc-prose")
|
||||||
|
expect(prose).to_have_count(1)
|
||||||
|
expect(prose).to_contain_text("Homelab network topology")
|
||||||
|
expect(prose).to_contain_text("rack-1 switch")
|
||||||
|
|
||||||
|
# The Full page escape hatch: the same graphic, full column width.
|
||||||
|
href = page.locator("#doc-modal-open").get_attribute("href")
|
||||||
|
assert href is not None and "/document.html" in href
|
||||||
|
page.goto(app_url + href)
|
||||||
|
viewer_img = page.locator(".doc-image-img--vector")
|
||||||
|
expect(viewer_img).to_be_visible(timeout=15_000)
|
||||||
|
box = viewer_img.bounding_box()
|
||||||
|
assert box is not None and box["width"] >= 600, (
|
||||||
|
f"the viewBox-only SVG did not span the viewer: {box!r}"
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# 2. The chat figure + the click-through preview in the RAG tab
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
def test_chat_figure_and_modal_preview(
|
||||||
|
page: Page, app_url: str, mock_llm: int, scratch: Path
|
||||||
|
) -> None:
|
||||||
|
"""TODO.md L3's citation half: the grounded question whose answer
|
||||||
|
cites the SVG doc (the scripted read echoes the read result — the
|
||||||
|
``IMAGE_DOC_MARKER`` line + the extracted words) shows the compact
|
||||||
|
``.source-image`` figure whose ``<img>`` LOADS from the bytes route
|
||||||
|
with the caption settling to the extracted text; clicking the
|
||||||
|
figure opens the SAME-page modal showing the full graphic (the
|
||||||
|
"click an svg … see a preview in the RAG tab rather than the xml"
|
||||||
|
contract) and Escape closes it."""
|
||||||
|
_seed_page(page, app_url, mock_llm, scratch)
|
||||||
|
doc = _doc(SOURCE, TOPO_PATH)
|
||||||
|
|
||||||
|
login(page, app_url, next="/")
|
||||||
|
expect(page.locator("#kb-banner")).to_be_hidden()
|
||||||
|
page.fill("#message-input", QUESTION)
|
||||||
|
page.click("#send-btn")
|
||||||
|
|
||||||
|
bubble = page.locator(".msg.brain .bubble").last
|
||||||
|
expect(bubble).to_contain_text(IMAGE_DOC_MARKER, timeout=30_000)
|
||||||
|
expect(bubble).to_contain_text("Homelab network topology", timeout=30_000)
|
||||||
|
expect(page.locator("#send-btn")).to_be_enabled(timeout=30_000)
|
||||||
|
expect(page.locator(".msg.brain.is-deflected")).to_have_count(0)
|
||||||
|
|
||||||
|
# The citation figure (phase 130: the figure IS the citation).
|
||||||
|
# Scoped by the figure's aria-label (the source/path label) — the
|
||||||
|
# seed also carries blank.svg, and an image doc the seed surfaces
|
||||||
|
# is auto-cited too (phase 124); the ASSERTED figure is the
|
||||||
|
# topology doc's.
|
||||||
|
row = page.locator(".msg.brain .msg-meta.msg-images")
|
||||||
|
expect(row).to_have_count(1)
|
||||||
|
fig = row.locator(f'.source-image[aria-label="{SOURCE}/{TOPO_PATH}"]')
|
||||||
|
expect(fig).to_have_count(1)
|
||||||
|
fig_img = fig.locator(".source-image-img")
|
||||||
|
expect(fig_img).to_be_visible(timeout=15_000)
|
||||||
|
assert fig_img.evaluate("el => el.naturalWidth") > 0, (
|
||||||
|
"the chat figure did not load the SVG bytes"
|
||||||
|
)
|
||||||
|
assert fig_img.get_attribute("src") == f"/api/documents/{doc.id}/image"
|
||||||
|
caption = fig.locator(".source-image-caption")
|
||||||
|
expect(caption).to_contain_text("Homelab network topology", timeout=15_000)
|
||||||
|
|
||||||
|
# The click → modal preview: the full graphic, no XML, and Escape.
|
||||||
|
fig.click()
|
||||||
|
modal = page.locator("#doc-modal")
|
||||||
|
expect(modal).to_be_visible(timeout=30_000)
|
||||||
|
modal_img = modal.locator("img.doc-image-img--vector")
|
||||||
|
expect(modal_img).to_have_count(1)
|
||||||
|
expect(modal_img).to_be_visible(timeout=15_000)
|
||||||
|
assert modal_img.evaluate("el => el.naturalWidth") > 0
|
||||||
|
assert "<svg" not in modal.inner_text()
|
||||||
|
page.keyboard.press("Escape")
|
||||||
|
expect(modal).to_be_hidden()
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# 3. The textless drawing: indexed with the structural line, previewable
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
def test_blank_svg_is_indexed_and_searchable(
|
||||||
|
page: Page, app_url: str, mock_llm: int, scratch: Path
|
||||||
|
) -> None:
|
||||||
|
"""LOCKED A2: ``diagrams/blank.svg`` carries NO words — pure paths —
|
||||||
|
yet it is NOT skipped: it exists as an image document whose indexed
|
||||||
|
text is the structural line built only from facts in the file
|
||||||
|
(``Vector diagram "blank" — <N> shapes (…), viewBox 640×480``), the
|
||||||
|
line is embedded (searchable), and the Sources page lists the doc
|
||||||
|
with its rendered thumbnail; the modal shows the graphic with that
|
||||||
|
line as the description below it."""
|
||||||
|
_seed_page(page, app_url, mock_llm, scratch)
|
||||||
|
doc = _doc(SOURCE, BLANK_PATH)
|
||||||
|
assert doc.is_image is True and doc.image_path is not None
|
||||||
|
assert doc.content.startswith('Vector diagram "blank" —')
|
||||||
|
assert "viewBox 640×480" in doc.content
|
||||||
|
assert doc.summary == doc.content
|
||||||
|
# Searchable: the structural line is an EMBEDDED chunk (both the
|
||||||
|
# content chunk and the is_summary mirror).
|
||||||
|
with SessionLocal() as db:
|
||||||
|
chunks = db.scalars(select(Chunk).where(Chunk.document_id == doc.id)).all()
|
||||||
|
assert [c.content for c in chunks] and all(
|
||||||
|
c.content == doc.content for c in chunks
|
||||||
|
)
|
||||||
|
assert all(c.embedding is not None and len(c.embedding) == 768 for c in chunks)
|
||||||
|
|
||||||
|
login(page, app_url)
|
||||||
|
_drill(page, SOURCE)
|
||||||
|
_drill(page, "diagrams")
|
||||||
|
row = page.locator("#docs-tbody tr", has_text=BLANK_PATH)
|
||||||
|
expect(row).to_have_count(1)
|
||||||
|
thumb = row.locator(".kb-doc-thumb-img")
|
||||||
|
expect(thumb).to_have_count(1)
|
||||||
|
expect(thumb).to_be_visible(timeout=15_000)
|
||||||
|
assert thumb.evaluate("el => el.naturalWidth") > 0, "the thumbnail is not painted"
|
||||||
|
|
||||||
|
# The modal: the graphic + the honest structural description (the
|
||||||
|
# file's only possible words).
|
||||||
|
row.locator("td:nth-child(2) a.doc-link").click()
|
||||||
|
modal = page.locator("#doc-modal")
|
||||||
|
expect(modal).to_be_visible(timeout=30_000)
|
||||||
|
expect(modal.locator("img.doc-image-img--vector")).to_have_count(1)
|
||||||
|
expect(modal.locator(".doc-prose")).to_contain_text('Vector diagram "blank"')
|
||||||
|
expect(modal.locator("pre.doc-raw")).to_have_count(0)
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# 4. The phase-180 synergy:  paints
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
def test_markdown_image_ref_renders(
|
||||||
|
page: Page, app_url: str, mock_llm: int, scratch: Path
|
||||||
|
) -> None:
|
||||||
|
"""The two-stage seeding the phases' hand-off needs: import the tree,
|
||||||
|
read the SVG doc's id from the DB, THEN write ``notes/inline.md``
|
||||||
|
containing ```` and re-import
|
||||||
|
(one doc added, the SVG row untouched — the fast path). The viewer
|
||||||
|
for that document renders ``img.md-img`` pointing at the bytes route
|
||||||
|
and LOADING it (``naturalWidth > 0``) — phase 180's allowlist
|
||||||
|
accepts the route shape, phase 181's route answers it, and the
|
||||||
|
reference never prints as text."""
|
||||||
|
_seed_page(page, app_url, mock_llm, scratch)
|
||||||
|
topo = _doc(SOURCE, TOPO_PATH)
|
||||||
|
image_url = f"/api/documents/{topo.id}/image"
|
||||||
|
assert extract_svg_text # the extractor produced the row's text (premise)
|
||||||
|
|
||||||
|
tree = scratch / "src" / SOURCE
|
||||||
|
(tree / "notes" / "inline.md").write_text(
|
||||||
|
"# Inline reference notes\n\n"
|
||||||
|
"The topology drawing, inline:\n\n"
|
||||||
|
f"\n\n"
|
||||||
|
"Same route, inside a link: [the diagram](/sources.html).\n",
|
||||||
|
encoding="utf-8",
|
||||||
|
)
|
||||||
|
second = _run_in_thread(_import(tree, mock_llm, scratch))
|
||||||
|
assert second.added == 1, second
|
||||||
|
assert second.svg_indexed == 0, "the unchanged SVG must take the fast path"
|
||||||
|
|
||||||
|
modal = _open_doc_modal(page, app_url, "notes", "notes/inline.md")
|
||||||
|
md_img = modal.locator("img.md-img")
|
||||||
|
expect(md_img).to_have_count(1)
|
||||||
|
assert md_img.get_attribute("src") == image_url
|
||||||
|
assert md_img.get_attribute("alt") == "topology"
|
||||||
|
expect(md_img).to_be_visible(timeout=15_000)
|
||||||
|
assert md_img.evaluate("el => el.naturalWidth") > 0, (
|
||||||
|
"the bytes route did not answer the markdown reference"
|
||||||
|
)
|
||||||
|
# The reference renders — it never prints.
|
||||||
|
assert "
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# 5. The smoke guard: raster image docs are byte-unchanged
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
def test_raster_surfaces_unchanged(
|
||||||
|
page: Page, app_url: str, mock_llm: int, scratch: Path
|
||||||
|
) -> None:
|
||||||
|
"""Phase 122/124 behaviour survives the vector branch: a PNG doc in
|
||||||
|
the SAME tree is described through the vision seam (the mock's
|
||||||
|
fixed answer), renders through the plain ``.doc-image-img`` with NO
|
||||||
|
``--vector`` class, and its bytes route carries NO sandbox CSP — the
|
||||||
|
app-wide middleware policy only, the pre-phase header set."""
|
||||||
|
_seed_page(page, app_url, mock_llm, scratch)
|
||||||
|
tree = scratch / "src" / SOURCE
|
||||||
|
(tree / "diagrams" / "raster.png").write_bytes(PNG_1X1)
|
||||||
|
summary = _run_in_thread(_import(tree, mock_llm, scratch))
|
||||||
|
assert (summary.files, summary.added) == (4, 1)
|
||||||
|
assert summary.svg_indexed == 0
|
||||||
|
|
||||||
|
doc = _doc(SOURCE, "diagrams/raster.png")
|
||||||
|
assert doc.is_image is True
|
||||||
|
image_url = f"/api/documents/{doc.id}/image"
|
||||||
|
|
||||||
|
login(page, app_url)
|
||||||
|
_drill(page, SOURCE)
|
||||||
|
_drill(page, "diagrams")
|
||||||
|
row = page.locator("#docs-tbody tr", has_text="diagrams/raster.png")
|
||||||
|
expect(row).to_have_count(1)
|
||||||
|
row.locator("td:nth-child(2) a.doc-link").click()
|
||||||
|
modal = page.locator("#doc-modal")
|
||||||
|
expect(modal).to_be_visible(timeout=30_000)
|
||||||
|
|
||||||
|
img = modal.locator(".doc-image-img")
|
||||||
|
expect(img).to_have_count(1)
|
||||||
|
expect(img).to_be_visible(timeout=15_000)
|
||||||
|
# The raster class string is UNTOUCHED by the vector modifier.
|
||||||
|
assert (img.get_attribute("class") or "") == "doc-image-img"
|
||||||
|
expect(modal.locator("img.doc-image-img--vector")).to_have_count(0)
|
||||||
|
|
||||||
|
resp = page.request.get(app_url + image_url)
|
||||||
|
assert resp.status == 200
|
||||||
|
assert resp.headers["content-type"] == "image/png"
|
||||||
|
csp = resp.headers.get("content-security-policy", "")
|
||||||
|
assert "sandbox" not in csp, "the SVG sandbox policy leaked onto a raster route"
|
||||||
+18
@@ -0,0 +1,18 @@
|
|||||||
|
<?xml version="1.0" encoding="UTF-8"?>
|
||||||
|
<!--
|
||||||
|
Phase 181 fixture: the TEXTLESS drawing (the LOCKED A2 class) — pure
|
||||||
|
paths and rects, no <title>, no <desc>, no <text>. The extractor's
|
||||||
|
structural fallback line is the only text this document can ever
|
||||||
|
carry, built from facts in the file: the stem, the shape census and
|
||||||
|
the viewBox. The phase's E2E pins the indexed content to
|
||||||
|
Vector diagram "blank" — … and the Sources thumbnail to the graphic.
|
||||||
|
-->
|
||||||
|
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 640 480">
|
||||||
|
<rect x="20" y="20" width="600" height="440" fill="none" stroke="#455a64" stroke-width="3"/>
|
||||||
|
<path d="M80 120 L240 120 L240 240 L80 240 Z" fill="#cfd8dc" stroke="#455a64" stroke-width="3"/>
|
||||||
|
<path d="M400 120 L560 120 L560 240 L400 240 Z" fill="#cfd8dc" stroke="#455a64" stroke-width="3"/>
|
||||||
|
<path d="M240 180 H400" stroke="#0288d1" stroke-width="4"/>
|
||||||
|
<path d="M320 240 V360" stroke="#0288d1" stroke-width="4"/>
|
||||||
|
<path d="M200 360 H440" stroke="#0288d1" stroke-width="4"/>
|
||||||
|
<rect x="280" y="340" width="80" height="60" fill="#eceff1" stroke="#455a64" stroke-width="3"/>
|
||||||
|
</svg>
|
||||||
|
After Width: | Height: | Size: 1.1 KiB |
+37
@@ -0,0 +1,37 @@
|
|||||||
|
<?xml version="1.0" encoding="UTF-8"?>
|
||||||
|
<!--
|
||||||
|
Phase 181 fixture: a hand-written homelab wiring diagram.
|
||||||
|
|
||||||
|
Deliberate properties (the phase's E2E contract, tests/e2e/
|
||||||
|
test_svg_document_preview.py):
|
||||||
|
* a viewBox and NO width/height attributes — a vector drawing with no
|
||||||
|
intrinsic size, the exact class the phase-181 vector sizing class
|
||||||
|
(`doc-image-img` plus its `vector` modifier) exists for — the
|
||||||
|
300x150 default box would sit small inside the full-width card;
|
||||||
|
* its words live in <title>, <desc> and two <text> labels — what
|
||||||
|
app.rag.svg_text.extract_svg_text indexes (LOCKED A1, no vision call);
|
||||||
|
* a dozen plain <path>/<rect> shapes give the drawing real geometry
|
||||||
|
while the title/desc/text carry the searchable words.
|
||||||
|
-->
|
||||||
|
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 1200 800">
|
||||||
|
<title>Homelab network topology</title>
|
||||||
|
<desc>Wiring diagram of the homelab: the edge router uplinks two managed switches, the hypervisor rack and the nas storage shelf.</desc>
|
||||||
|
<rect x="40" y="40" width="1120" height="720" fill="none" stroke="#37474f" stroke-width="4"/>
|
||||||
|
<rect x="120" y="96" width="240" height="120" rx="12" fill="#eceff1" stroke="#37474f" stroke-width="3"/>
|
||||||
|
<path d="M160 156 h160 M240 126 v60" stroke="#546e7a" stroke-width="5"/>
|
||||||
|
<rect x="480" y="96" width="240" height="120" rx="12" fill="#eceff1" stroke="#37474f" stroke-width="3"/>
|
||||||
|
<path d="M520 136 h160 M520 166 h160 M520 196 h160" stroke="#546e7a" stroke-width="4"/>
|
||||||
|
<rect x="840" y="96" width="240" height="120" rx="12" fill="#eceff1" stroke="#37474f" stroke-width="3"/>
|
||||||
|
<path d="M880 136 h160 M880 166 h160 M880 196 h160" stroke="#546e7a" stroke-width="4"/>
|
||||||
|
<path d="M360 156 C 420 156 420 156 480 156" fill="none" stroke="#0288d1" stroke-width="5"/>
|
||||||
|
<path d="M720 156 C 780 156 780 156 840 156" fill="none" stroke="#0288d1" stroke-width="5"/>
|
||||||
|
<path d="M600 216 C 600 300 600 340 600 400" fill="none" stroke="#0288d1" stroke-width="5"/>
|
||||||
|
<rect x="360" y="400" width="240" height="140" rx="12" fill="#e8f5e9" stroke="#2e7d32" stroke-width="3"/>
|
||||||
|
<path d="M400 440 h160 M400 470 h160 M400 500 h160" stroke="#2e7d32" stroke-width="4"/>
|
||||||
|
<rect x="680" y="400" width="240" height="140" rx="12" fill="#fff3e0" stroke="#ef6c00" stroke-width="3"/>
|
||||||
|
<path d="M720 440 h160 M720 470 h160 M720 500 h160" stroke="#ef6c00" stroke-width="4"/>
|
||||||
|
<path d="M480 540 C 480 620 600 620 600 620 M720 540 C 720 620 600 620 600 620" fill="none" stroke="#78909c" stroke-width="4"/>
|
||||||
|
<path d="M600 620 v100" fill="none" stroke="#78909c" stroke-width="4"/>
|
||||||
|
<text x="180" y="250" font-family="sans-serif" font-size="34" fill="#37474f">rack-1 switch</text>
|
||||||
|
<text x="700" y="590" font-family="sans-serif" font-size="34" fill="#ef6c00">nas storage</text>
|
||||||
|
</svg>
|
||||||
|
After Width: | Height: | Size: 2.7 KiB |
+14
@@ -0,0 +1,14 @@
|
|||||||
|
# Homelab infrastructure notes
|
||||||
|
|
||||||
|
The wiring diagram `diagrams/topology.svg` — titled "Homelab network
|
||||||
|
topology" — is the current-state reference for the rack layout. When a
|
||||||
|
question is about the homelab network topology, that diagram is the
|
||||||
|
picture to open: it shows the edge router uplinking two managed
|
||||||
|
switches, the hypervisor rack and the nas storage shelf.
|
||||||
|
|
||||||
|
Uplink rules, restated in prose so the words are searchable next to the
|
||||||
|
diagram: the edge router hands one 10Gb link to rack-1 switch and one
|
||||||
|
to the second switch; the hypervisor VLANs live on rack-1 switch, and
|
||||||
|
nas storage sits behind the second switch on its own segment. The
|
||||||
|
diagram carries no other labels — anything else about the network
|
||||||
|
belongs in this note, not in the drawing.
|
||||||
@@ -0,0 +1,443 @@
|
|||||||
|
"""Integration tests: phase 181 (task 03) — the SVG half of the image
|
||||||
|
BYTES route ``GET /api/documents/{doc_id}/image``.
|
||||||
|
|
||||||
|
The contract this module pins (LOCKED A3):
|
||||||
|
|
||||||
|
* an ``is_image`` row whose stored copy is an ``.svg`` serves
|
||||||
|
``Content-Type: image/svg+xml`` (the ``app.rag.summarizer.IMAGE_MIMES``
|
||||||
|
map — one map, one truth with the describe call's data-URL mime) plus
|
||||||
|
ONE extra response header, ``Content-Security-Policy: default-src
|
||||||
|
'none'; sandbox``. An ``<img>`` can never run an SVG's script, but the
|
||||||
|
URL is DIRECTLY navigable, and the bytes come from an untrusted source
|
||||||
|
tree — so a direct visit must not hand the browser a script-capable
|
||||||
|
SAME-ORIGIN document. The route sets the policy itself and
|
||||||
|
``SecurityHeadersMiddleware`` PRESERVES a CSP an inner layer already
|
||||||
|
set, so this is what the browser sees; the middleware's global
|
||||||
|
``X-Content-Type-Options: nosniff`` and ``X-Frame-Options: DENY`` still
|
||||||
|
land on the response;
|
||||||
|
* the SVG bytes are served VERBATIM (no rewriting, no
|
||||||
|
``Content-Disposition: attachment`` — the graphic must render inline
|
||||||
|
and the policy is the control, not a sanitizer);
|
||||||
|
* every RASTER response is byte-identical to its pre-phase shape — the
|
||||||
|
same header KEY set, the app-wide CSP from the middleware (no
|
||||||
|
``sandbox``, no route-specific policy), the same ``Cache-Control`` and
|
||||||
|
the same exact bytes;
|
||||||
|
* the phase-122 posture is unchanged: PUBLIC (phase 124 LOCKED A4 — the
|
||||||
|
unguessable ``uuid4`` id IS the credential, the shared chat shows the
|
||||||
|
figure to a guest) with the four 404 ``document not found`` shapes
|
||||||
|
(unknown id, unparseable id — 404 and never 422, a text doc, a row
|
||||||
|
whose copy is gone from disk).
|
||||||
|
|
||||||
|
Task 05 of this phase adds the INGESTION-TO-SERVE round trip: the
|
||||||
|
task-03 content contract on ``GET /api/documents/content`` for an SVG
|
||||||
|
row (``is_image: true``, an ``image_url`` of the form
|
||||||
|
``/api/documents/<uuid>/image``, ``format: "svg"`` and the extracted
|
||||||
|
``<title>``/``<desc>``/``<text>`` words as ``content``) and the
|
||||||
|
WIRE-CONTRACT PIN that ties the phases together: that literal
|
||||||
|
``image_url`` is accepted by phase 180's ``allowImageUrl`` renderer
|
||||||
|
allowlist — the ```` synergy (an answer or a
|
||||||
|
markdown document referencing the bytes route paints the graphic).
|
||||||
|
|
||||||
|
Uses the real compose Postgres (``db`` fixture) and FastAPI's TestClient.
|
||||||
|
"""
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import hashlib
|
||||||
|
import re
|
||||||
|
import uuid
|
||||||
|
from collections.abc import Iterator
|
||||||
|
from datetime import UTC, datetime
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
from fastapi.testclient import TestClient
|
||||||
|
from sqlalchemy import text
|
||||||
|
|
||||||
|
import app.api.docs as docs_api
|
||||||
|
from app.core.security_headers import CSP
|
||||||
|
from app.main import app as fastapi_app
|
||||||
|
from app.models import Chunk, Document
|
||||||
|
from app.rag.summarizer import IMAGE_MIMES
|
||||||
|
from app.rag.svg_text import extract_svg_text
|
||||||
|
|
||||||
|
#: The phase-180 shared renderer — the file whose ``allowImageUrl``
|
||||||
|
#: allowlist decides whether a ```` reference paints an
|
||||||
|
#: ``img.md-img`` or stays inert text (the wire-contract pin below).
|
||||||
|
MARKDOWN_JS = Path(__file__).resolve().parents[2] / "frontend" / "assets" / "markdown.js"
|
||||||
|
|
||||||
|
#: The 1×1 PNG the raster control row stores (the ``test_docs_api.py``
|
||||||
|
#: fixture's bytes — a real PNG header so a Content-Type mix-up is
|
||||||
|
#: visible in the value, not just in the length).
|
||||||
|
PNG_1X1 = bytes.fromhex(
|
||||||
|
"89504e470d0a1a0a0000000d49484452000000010000000108060000001f15c489"
|
||||||
|
"0000000d4944415478da63fcffff3f0300050201000d5ca5e90000000049454e44ae426082"
|
||||||
|
)
|
||||||
|
|
||||||
|
#: One SVG carrying BOTH its own words and a ``<script>`` sentinel. The
|
||||||
|
#: script is what makes the pin meaningful: the route does NOT sanitize
|
||||||
|
#: (it streams the stored bytes untouched — the viewer needs the real
|
||||||
|
#: graphic), so the ONLY thing standing between a stored SVG and a
|
||||||
|
#: script-capable document is the policy this module pins.
|
||||||
|
SVG_WITH_SCRIPT = (
|
||||||
|
b'<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 120 80">'
|
||||||
|
b"<title>Topology</title><desc>Two nodes, one link</desc>"
|
||||||
|
b'<text x="8" y="12">edge router</text>'
|
||||||
|
b"<path d='M10 60 L110 20'/>"
|
||||||
|
b"<script>alert('probe')</script></svg>"
|
||||||
|
)
|
||||||
|
|
||||||
|
#: The response-header KEY set both the raster and the vector serve share
|
||||||
|
#: (FileResponse's own stat headers + the route's Cache-Control + the
|
||||||
|
#: three the SecurityHeadersMiddleware adds). Pinned so a future change
|
||||||
|
#: cannot silently drop or add a header on either branch — the vector
|
||||||
|
#: branch differs from the raster branch in the VALUE of one key, never
|
||||||
|
#: in the key set.
|
||||||
|
SERVE_HEADER_KEYS = {
|
||||||
|
"accept-ranges",
|
||||||
|
"cache-control",
|
||||||
|
"content-length",
|
||||||
|
"content-security-policy",
|
||||||
|
"content-type",
|
||||||
|
"etag",
|
||||||
|
"last-modified",
|
||||||
|
"x-content-type-options",
|
||||||
|
"x-frame-options",
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def _cleanup_kb(db) -> None:
|
||||||
|
"""Truncate the KB tables (commit first — SQLAlchemy does NOT
|
||||||
|
autoflush pending ORM objects before a raw ``text()`` statement, and
|
||||||
|
a pending chunk INSERT flushed after the TRUNCATE would FK-violate)."""
|
||||||
|
db.commit()
|
||||||
|
db.execute(text("TRUNCATE chunks, documents"))
|
||||||
|
db.commit()
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.fixture()
|
||||||
|
def kb(db) -> Iterator:
|
||||||
|
"""The real Postgres with the KB tables emptied before and after the
|
||||||
|
test (the module's own rows are the only ones present)."""
|
||||||
|
_cleanup_kb(db)
|
||||||
|
yield db
|
||||||
|
_cleanup_kb(db)
|
||||||
|
|
||||||
|
|
||||||
|
def _seed_image_row(
|
||||||
|
db,
|
||||||
|
tmp_path: Path,
|
||||||
|
*,
|
||||||
|
source: str = "SvgSrc",
|
||||||
|
path: str = "diagram.svg",
|
||||||
|
data: bytes = SVG_WITH_SCRIPT,
|
||||||
|
content: str = "Topology — Two nodes, one link — edge router",
|
||||||
|
image_path: str | None = "auto",
|
||||||
|
) -> Document:
|
||||||
|
"""One ``is_image`` document row plus its persistent copy (the
|
||||||
|
importer's ``image_dir`` layout — ``<image_dir>/<doc-id><ext>``)
|
||||||
|
under *tmp_path*. ``image_path="auto"`` writes the copy at the
|
||||||
|
importer's layout; ``None`` leaves the row without a copy path; any
|
||||||
|
other string pins the column verbatim (the lost-copy corner writes
|
||||||
|
the copy and unlinks it in the test itself)."""
|
||||||
|
doc = Document(
|
||||||
|
id=uuid.uuid4(),
|
||||||
|
source=source,
|
||||||
|
path=path,
|
||||||
|
full_path=f"/tmp/{path}",
|
||||||
|
title=Path(path).stem,
|
||||||
|
content=content,
|
||||||
|
content_hash=hashlib.sha256(data).hexdigest(),
|
||||||
|
indexed_at=datetime.now(UTC),
|
||||||
|
created_at=datetime(2026, 1, 1, tzinfo=UTC),
|
||||||
|
summary=content,
|
||||||
|
is_image=True,
|
||||||
|
)
|
||||||
|
if image_path == "auto":
|
||||||
|
copy = tmp_path / f"{doc.id}{Path(path).suffix}"
|
||||||
|
copy.write_bytes(data)
|
||||||
|
doc.image_path = str(copy)
|
||||||
|
elif image_path is not None:
|
||||||
|
doc.image_path = image_path
|
||||||
|
db.add(doc)
|
||||||
|
db.flush()
|
||||||
|
db.add_all(
|
||||||
|
[
|
||||||
|
Chunk(
|
||||||
|
document_id=doc.id, position=0, content=content, embedding=[0.01] * 768
|
||||||
|
),
|
||||||
|
Chunk(
|
||||||
|
document_id=doc.id,
|
||||||
|
position=-1,
|
||||||
|
content=content,
|
||||||
|
embedding=[0.01] * 768,
|
||||||
|
is_summary=True,
|
||||||
|
),
|
||||||
|
]
|
||||||
|
)
|
||||||
|
db.commit()
|
||||||
|
return doc
|
||||||
|
|
||||||
|
|
||||||
|
def _seed_text_row(db, *, source: str = "TextSrc", path: str = "note.md") -> uuid.UUID:
|
||||||
|
"""One plain TEXT document (never ``is_image``) — a 404 shape."""
|
||||||
|
doc = Document(
|
||||||
|
id=uuid.uuid4(),
|
||||||
|
source=source,
|
||||||
|
path=path,
|
||||||
|
full_path=f"/tmp/{path}",
|
||||||
|
title=Path(path).stem,
|
||||||
|
content="# Note\n\nBody.",
|
||||||
|
content_hash=hashlib.sha256(path.encode("utf-8")).hexdigest(),
|
||||||
|
indexed_at=datetime.now(UTC),
|
||||||
|
created_at=datetime(2026, 1, 1, tzinfo=UTC),
|
||||||
|
)
|
||||||
|
db.add(doc)
|
||||||
|
db.commit()
|
||||||
|
return doc.id
|
||||||
|
|
||||||
|
|
||||||
|
# --- the vector serve contract ------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
def test_svg_row_serves_the_vector_mime_with_the_sandboxed_csp(
|
||||||
|
kb, tmp_path: Path, admin_client: TestClient
|
||||||
|
) -> None:
|
||||||
|
"""An SVG row → 200 ``image/svg+xml`` + the sandboxed CSP + the
|
||||||
|
unchanged Cache-Control, and the body is byte-equal to the stored
|
||||||
|
copy (the script sentinel included — the route is a transport, not a
|
||||||
|
sanitizer; ``sandbox`` blocks scripting and makes the document an
|
||||||
|
opaque origin, ``default-src 'none'`` blocks every subresource)."""
|
||||||
|
doc = _seed_image_row(kb, tmp_path)
|
||||||
|
assert doc.image_path is not None
|
||||||
|
|
||||||
|
r = admin_client.get(f"/api/documents/{doc.id}/image")
|
||||||
|
|
||||||
|
assert r.status_code == 200
|
||||||
|
assert r.headers["content-type"] == "image/svg+xml"
|
||||||
|
# LOCKED A3, verbatim.
|
||||||
|
assert r.headers["content-security-policy"] == "default-src 'none'; sandbox"
|
||||||
|
assert r.headers["cache-control"] == "private, max-age=3600"
|
||||||
|
assert r.headers["x-content-type-options"] == "nosniff"
|
||||||
|
assert r.headers["x-frame-options"] == "DENY"
|
||||||
|
assert r.content == SVG_WITH_SCRIPT # the EXACT stored bytes, nothing stripped
|
||||||
|
assert Path(doc.image_path).read_bytes() == r.content
|
||||||
|
|
||||||
|
|
||||||
|
def test_svg_route_header_key_set_is_the_serve_surface(
|
||||||
|
kb, tmp_path: Path, admin_client: TestClient
|
||||||
|
) -> None:
|
||||||
|
"""The vector response's header KEY set equals the shared serve
|
||||||
|
surface — one extra VALUE (the CSP), no extra header, and no
|
||||||
|
``content-disposition`` (the graphic must render inline)."""
|
||||||
|
doc = _seed_image_row(kb, tmp_path)
|
||||||
|
|
||||||
|
r = admin_client.get(f"/api/documents/{doc.id}/image")
|
||||||
|
|
||||||
|
assert set(r.headers) == SERVE_HEADER_KEYS
|
||||||
|
assert "content-disposition" not in r.headers
|
||||||
|
|
||||||
|
|
||||||
|
def test_svg_mime_guard_is_derived_from_the_map_never_from_the_client(
|
||||||
|
kb, tmp_path: Path, admin_client: TestClient
|
||||||
|
) -> None:
|
||||||
|
"""The guard keys on the resolved ``media_type`` — the extension map
|
||||||
|
(lower-cased, from the ROW's stored path), never a client value: an
|
||||||
|
upper-case ``.SVG`` copy is still vector, and an extension the map
|
||||||
|
does not name still takes the generic fallback with NO sandbox."""
|
||||||
|
assert docs_api.SVG_MEDIA_TYPE == IMAGE_MIMES[".svg"] == "image/svg+xml"
|
||||||
|
assert docs_api.SVG_SANDBOX_CSP == "default-src 'none'; sandbox"
|
||||||
|
|
||||||
|
upper = _seed_image_row(kb, tmp_path, path="diagram.SVG", source="UpperSrc")
|
||||||
|
assert upper.image_path is not None
|
||||||
|
upper_copy = Path(upper.image_path)
|
||||||
|
upper_copy.unlink()
|
||||||
|
upper_copy = upper_copy.with_name(f"{upper.id}.SVG")
|
||||||
|
upper_copy.write_bytes(SVG_WITH_SCRIPT)
|
||||||
|
upper.image_path = str(upper_copy)
|
||||||
|
kb.commit()
|
||||||
|
|
||||||
|
r = admin_client.get(f"/api/documents/{upper.id}/image")
|
||||||
|
assert r.status_code == 200
|
||||||
|
assert r.headers["content-type"] == "image/svg+xml"
|
||||||
|
assert r.headers["content-security-policy"] == docs_api.SVG_SANDBOX_CSP
|
||||||
|
|
||||||
|
unknown = _seed_image_row(
|
||||||
|
kb, tmp_path, path="mystery.bin", source="UnknownSrc", data=PNG_1X1
|
||||||
|
)
|
||||||
|
r = admin_client.get(f"/api/documents/{unknown.id}/image")
|
||||||
|
assert r.status_code == 200
|
||||||
|
assert r.headers["content-type"] == "application/octet-stream"
|
||||||
|
# No sandbox — the policy is scoped to the vector type only.
|
||||||
|
assert r.headers["content-security-policy"] == CSP
|
||||||
|
|
||||||
|
|
||||||
|
# --- the raster branch is untouched -------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
def test_raster_row_response_headers_are_unchanged_by_the_svg_rule(
|
||||||
|
kb, tmp_path: Path, admin_client: TestClient
|
||||||
|
) -> None:
|
||||||
|
"""A PNG row is served EXACTLY as before phase 181: ``image/png``,
|
||||||
|
the same Cache-Control, the same header KEY set, and a CSP equal to
|
||||||
|
the app-wide A20 string from ``SecurityHeadersMiddleware`` — no
|
||||||
|
route-specific policy, no ``sandbox`` anywhere in it. The byte
|
||||||
|
equality of the body is the raster half of "nothing changed"."""
|
||||||
|
doc = _seed_image_row(
|
||||||
|
kb,
|
||||||
|
tmp_path,
|
||||||
|
source="PngSrc",
|
||||||
|
path="pic.png",
|
||||||
|
data=PNG_1X1,
|
||||||
|
content="A red square on a white background.",
|
||||||
|
)
|
||||||
|
|
||||||
|
r = admin_client.get(f"/api/documents/{doc.id}/image")
|
||||||
|
|
||||||
|
assert r.status_code == 200
|
||||||
|
assert r.headers["content-type"] == "image/png"
|
||||||
|
assert r.headers["cache-control"] == "private, max-age=3600"
|
||||||
|
assert r.headers["content-security-policy"] == CSP
|
||||||
|
assert "sandbox" not in r.headers["content-security-policy"]
|
||||||
|
assert r.headers["x-content-type-options"] == "nosniff"
|
||||||
|
assert set(r.headers) == SERVE_HEADER_KEYS
|
||||||
|
assert r.content == PNG_1X1
|
||||||
|
|
||||||
|
|
||||||
|
# --- the public posture and the 404 shapes (unchanged) ------------------
|
||||||
|
|
||||||
|
|
||||||
|
def test_svg_bytes_are_public_with_the_sandbox(
|
||||||
|
kb, tmp_path: Path, admin_client: TestClient
|
||||||
|
) -> None:
|
||||||
|
"""Phase 124 LOCKED A4 stays intact for the vector type: a FRESH
|
||||||
|
client (no session — the shared chat's guest) gets the same bytes
|
||||||
|
and the same sandboxed CSP as a signed-in one. The policy is not an
|
||||||
|
access control (the unguessable ``uuid4`` id is), so making the
|
||||||
|
response safe does not make it private."""
|
||||||
|
doc = _seed_image_row(kb, tmp_path)
|
||||||
|
|
||||||
|
anonymous = TestClient(fastapi_app)
|
||||||
|
r = anonymous.get(f"/api/documents/{doc.id}/image")
|
||||||
|
assert r.status_code == 200
|
||||||
|
assert r.headers["content-type"] == "image/svg+xml"
|
||||||
|
assert r.headers["content-security-policy"] == "default-src 'none'; sandbox"
|
||||||
|
assert r.content == SVG_WITH_SCRIPT
|
||||||
|
|
||||||
|
r_admin = admin_client.get(f"/api/documents/{doc.id}/image")
|
||||||
|
assert r_admin.status_code == 200
|
||||||
|
assert r_admin.content == r.content
|
||||||
|
|
||||||
|
|
||||||
|
def test_svg_route_404_shapes(kb, tmp_path: Path, admin_client: TestClient) -> None:
|
||||||
|
"""Every non-servable case is still 404 ``document not found`` —
|
||||||
|
including the SVG rows: an unknown uuid, a NON-uuid string (404, not
|
||||||
|
422 — a guessed id is an unknown document), a TEXT document (no
|
||||||
|
bytes behind it), an image row whose ``image_path`` is NULL, and an
|
||||||
|
SVG row whose copy was deleted from disk."""
|
||||||
|
text_doc_id = _seed_text_row(kb)
|
||||||
|
no_copy = _seed_image_row(kb, tmp_path, path="nocopy.svg", image_path=None)
|
||||||
|
lost = _seed_image_row(kb, tmp_path, path="lost.svg", source="LostSrc")
|
||||||
|
assert lost.image_path is not None
|
||||||
|
Path(lost.image_path).unlink()
|
||||||
|
|
||||||
|
for doc_id in (
|
||||||
|
str(uuid.uuid4()), # unknown id
|
||||||
|
"not-a-uuid", # unparseable id → 404, never 422
|
||||||
|
str(text_doc_id), # text document
|
||||||
|
str(no_copy.id), # image row, image_path NULL
|
||||||
|
str(lost.id), # image row, copy gone from disk
|
||||||
|
):
|
||||||
|
r = admin_client.get(f"/api/documents/{doc_id}/image")
|
||||||
|
assert r.status_code == 404, doc_id
|
||||||
|
assert r.json() == {"detail": "document not found"}, doc_id
|
||||||
|
|
||||||
|
|
||||||
|
# --- the content endpoint on an SVG row (task 05, the ingestion shape) ---
|
||||||
|
|
||||||
|
|
||||||
|
def test_content_endpoint_serves_the_svg_row_as_a_vector_image(
|
||||||
|
kb, tmp_path: Path, admin_client: TestClient
|
||||||
|
) -> None:
|
||||||
|
"""``GET /api/documents/content`` for an SVG row carries the four
|
||||||
|
keys the viewer's vector branch needs: ``is_image: true``, an
|
||||||
|
``image_url`` of the exact ``/api/documents/<uuid>/image`` form
|
||||||
|
(the bytes route above), ``format: "svg"`` — the ONLY place in the
|
||||||
|
app that knows the type (task 04's ``doc-image-img--vector`` keys
|
||||||
|
off it) — and ``content`` equal to the SVG's OWN extracted
|
||||||
|
``<title>``/``<desc>``/``<text>`` words (LOCKED A1: the row's text
|
||||||
|
is what the importer stored, never raw XML re-derived here). The
|
||||||
|
``<script>`` sentinel in the stored bytes never reaches this
|
||||||
|
payload — the DB text IS the extraction."""
|
||||||
|
extracted = extract_svg_text(SVG_WITH_SCRIPT, stem="diagram")
|
||||||
|
assert extracted is not None and extracted.structural is False
|
||||||
|
doc = _seed_image_row(
|
||||||
|
kb, tmp_path, content=extracted.text, source="VecSrc", path="vector.svg"
|
||||||
|
)
|
||||||
|
|
||||||
|
r = admin_client.get(
|
||||||
|
"/api/documents/content", params={"source": "VecSrc", "path": "vector.svg"}
|
||||||
|
)
|
||||||
|
|
||||||
|
assert r.status_code == 200
|
||||||
|
body = r.json()
|
||||||
|
assert body["is_image"] is True
|
||||||
|
assert body["image_url"] == f"/api/documents/{doc.id}/image"
|
||||||
|
assert body["format"] == "svg"
|
||||||
|
assert body["content"] == extracted.text
|
||||||
|
assert body["content"] == "Topology\nTwo nodes, one link\nedge router"
|
||||||
|
assert body["summary"] == extracted.text
|
||||||
|
# The serve round trip: the image_url the payload advertises answers.
|
||||||
|
img = admin_client.get(body["image_url"])
|
||||||
|
assert img.status_code == 200
|
||||||
|
assert img.headers["content-type"] == "image/svg+xml"
|
||||||
|
assert img.content == SVG_WITH_SCRIPT
|
||||||
|
|
||||||
|
|
||||||
|
# --- the wire-contract pin: allowImageUrl accepts the route (task 05) ---
|
||||||
|
|
||||||
|
|
||||||
|
def _allow_image_url() -> re.Pattern[str]:
|
||||||
|
"""The phase-180 ``allowImageUrl`` regex, extracted verbatim from
|
||||||
|
``frontend/assets/markdown.js`` (the JS literal is valid Python
|
||||||
|
syntax — one source, no hand-mirrored copy). A failed extraction
|
||||||
|
fails loudly: the pin only means something if it is THE shipped
|
||||||
|
allowlist."""
|
||||||
|
js = MARKDOWN_JS.read_text(encoding="utf-8")
|
||||||
|
match = re.search(
|
||||||
|
r"function allowImageUrl\(url\) \{\s*return (/.*?/[a-z]*)\.test\(url\)",
|
||||||
|
js,
|
||||||
|
re.S,
|
||||||
|
)
|
||||||
|
assert match is not None, "allowImageUrl no longer has the shape this pin reads"
|
||||||
|
literal = match.group(1)
|
||||||
|
pattern, _, flags = literal[1:].rpartition("/")
|
||||||
|
assert set(flags) <= {"i"}, f"unexpected regex flags {flags!r}"
|
||||||
|
return re.compile(pattern, re.IGNORECASE if "i" in flags else 0)
|
||||||
|
|
||||||
|
|
||||||
|
def test_image_url_shape_passes_the_phase_180_image_allowlist(
|
||||||
|
kb, tmp_path: Path, admin_client: TestClient
|
||||||
|
) -> None:
|
||||||
|
"""The phases-tie-together pin: the literal ``image_url`` an SVG row
|
||||||
|
advertises passes the renderer's ``allowImageUrl`` — so an answer
|
||||||
|
(or a markdown document) that carries ```` paints
|
||||||
|
``img.md-img`` from THIS route instead of printing the reference
|
||||||
|
(phase 180 + phase 181 end to end; the E2E's
|
||||||
|
``test_markdown_image_ref_renders`` is the browser half). The
|
||||||
|
sanity probe proves the extracted regex actually discriminates:
|
||||||
|
the same allowlist rejects a ``javascript:`` src (LOCKED A5)."""
|
||||||
|
doc = _seed_image_row(kb, tmp_path, source="PinSrc", path="wire.svg")
|
||||||
|
|
||||||
|
r = admin_client.get(
|
||||||
|
"/api/documents/content", params={"source": "PinSrc", "path": "wire.svg"}
|
||||||
|
)
|
||||||
|
image_url = r.json()["image_url"]
|
||||||
|
assert image_url == f"/api/documents/{doc.id}/image"
|
||||||
|
# The literal path the renderer receives (no query, no fragment).
|
||||||
|
assert re.fullmatch(r"/api/documents/[0-9a-f-]{36}/image", image_url)
|
||||||
|
|
||||||
|
allow = _allow_image_url()
|
||||||
|
assert allow.search(image_url), f"allowImageUrl rejected its own route: {image_url}"
|
||||||
|
# The discriminating half: the allowlist is real, not "accept all".
|
||||||
|
assert allow.search("javascript:alert(1)") is None
|
||||||
|
assert allow.search("data:text/html,<script>") is None
|
||||||
@@ -68,6 +68,7 @@ from pydantic import ValidationError
|
|||||||
|
|
||||||
from app.api import chat as chat_api
|
from app.api import chat as chat_api
|
||||||
from app.api import chat_images
|
from app.api import chat_images
|
||||||
|
from app.api.chat_images import _VECTOR_IMAGE_EXTENSIONS
|
||||||
from app.config import Settings
|
from app.config import Settings
|
||||||
from app.main import app as fastapi_app
|
from app.main import app as fastapi_app
|
||||||
from app.models import Document
|
from app.models import Document
|
||||||
@@ -354,6 +355,32 @@ def test_upload_rejects_a_non_image_extension_with_the_fixed_detail(
|
|||||||
assert not image_dir.exists() # the dir is created on the accepted path only
|
assert not image_dir.exists() # the dir is created on the accepted path only
|
||||||
|
|
||||||
|
|
||||||
|
def test_upload_rejects_a_vector_svg_even_though_it_indexes(
|
||||||
|
client: TestClient, image_dir: Path
|
||||||
|
) -> None:
|
||||||
|
"""Phase 181: ``svg`` joins the DEFAULT image set for the DOCUMENT
|
||||||
|
walk, and question attachments stay RASTER-only — the upload 422s
|
||||||
|
with the six-extension detail and writes nothing.
|
||||||
|
|
||||||
|
Without the subtraction this would be a silent trap: the stored
|
||||||
|
``<uuid>.svg`` is refused by the serve route's filename regex (a
|
||||||
|
broken figure in the user's own bubble) and an OpenAI-compatible
|
||||||
|
vision endpoint rejects ``image/svg+xml``, so the turn could never
|
||||||
|
be answered. The vector family belongs to INDEXING, not to
|
||||||
|
attachments."""
|
||||||
|
assert ".svg" in Settings(_env_file=None).image_extension_set # pyright: ignore[reportCallIssue]
|
||||||
|
assert frozenset({".svg"}) == _VECTOR_IMAGE_EXTENSIONS
|
||||||
|
r = _upload(
|
||||||
|
client, "diagram.svg", b'<svg xmlns="http://www.w3.org/2000/svg"></svg>',
|
||||||
|
mime="image/svg+xml",
|
||||||
|
)
|
||||||
|
assert r.status_code == 422
|
||||||
|
assert r.json()["detail"] == (
|
||||||
|
"only .bmp, .gif, .jpeg, .jpg, .png, .webp images are accepted"
|
||||||
|
)
|
||||||
|
assert not image_dir.exists()
|
||||||
|
|
||||||
|
|
||||||
def test_upload_rejects_a_name_without_an_extension(
|
def test_upload_rejects_a_name_without_an_extension(
|
||||||
client: TestClient, image_dir: Path
|
client: TestClient, image_dir: Path
|
||||||
) -> None:
|
) -> None:
|
||||||
@@ -916,7 +943,10 @@ def test_the_attach_reveal_is_gated_on_the_config_flag() -> None:
|
|||||||
def test_the_client_precheck_matches_the_servers_six_extensions() -> None:
|
def test_the_client_precheck_matches_the_servers_six_extensions() -> None:
|
||||||
"""ONE list of the six, on both sides: the client pre-check (the
|
"""ONE list of the six, on both sides: the client pre-check (the
|
||||||
file NAME's extension — the accept attribute is advisory) and the
|
file NAME's extension — the accept attribute is advisory) and the
|
||||||
server's phase-122 frozenset (the upload endpoint's authority).
|
server's phase-122 frozenset MINUS the vector members (the upload
|
||||||
|
endpoint's authority — phase 181: question images stay RASTER-only,
|
||||||
|
a vision endpoint rejects ``image/svg+xml`` and the serve route's
|
||||||
|
filename regex would never serve one).
|
||||||
The client list must stay the code default's six — a drift here
|
The client list must stay the code default's six — a drift here
|
||||||
would either block a valid pick or waste a round-trip on a 422."""
|
would either block a valid pick or waste a round-trip on a 422."""
|
||||||
js = _app_js()
|
js = _app_js()
|
||||||
@@ -931,12 +961,14 @@ def test_the_client_precheck_matches_the_servers_six_extensions() -> None:
|
|||||||
"png",
|
"png",
|
||||||
"webp",
|
"webp",
|
||||||
}
|
}
|
||||||
# and it equals the server's default set (dots stripped — the
|
# and it equals the server's default raster set (dots stripped — the
|
||||||
# client list is dotless, the server's is dotted).
|
# client list is dotless, the server's is dotted). ``svg`` is in the
|
||||||
server_set = {
|
# DEFAULT image set for the document walk (phase 181) and
|
||||||
e.lstrip(".")
|
# deliberately NOT attachable here (see
|
||||||
for e in Settings(_env_file=None).image_extension_set # pyright: ignore[reportCallIssue]
|
# ``chat_images._VECTOR_IMAGE_EXTENSIONS``).
|
||||||
}
|
raw_default = Settings(_env_file=None).image_extension_set # pyright: ignore[reportCallIssue]
|
||||||
|
assert ".svg" in raw_default # the vector member (phase 181)
|
||||||
|
server_set = {e.lstrip(".") for e in raw_default - _VECTOR_IMAGE_EXTENSIONS}
|
||||||
assert client_set == server_set
|
assert client_set == server_set
|
||||||
# the file input's accept attribute names the same six (mime form).
|
# the file input's accept attribute names the same six (mime form).
|
||||||
m = re.search(r'<input type="file" id="attach-file"\s+accept="([^"]+)"', _index_html())
|
m = re.search(r'<input type="file" id="attach-file"\s+accept="([^"]+)"', _index_html())
|
||||||
|
|||||||
@@ -94,6 +94,20 @@ format table as node behavioral runs over the real module, and the
|
|||||||
``.doc-prose`` CSS (the 72rem reading column + ``pre-wrap`` +
|
``.doc-prose`` CSS (the 72rem reading column + ``pre-wrap`` +
|
||||||
``overflow-wrap: anywhere``, tokens only) beside the ``.doc-raw``
|
``overflow-wrap: anywhere``, tokens only) beside the ``.doc-raw``
|
||||||
contract that stays locked (phase 116 A1 / phase 165 A3).
|
contract that stays locked (phase 116 A1 / phase 165 A3).
|
||||||
|
|
||||||
|
Phase 181 (task 04) — the SVG vector preview: ``docImageBlock`` adds
|
||||||
|
the ``doc-image-img--vector`` modifier ONLY when
|
||||||
|
``doc.format === "svg"`` (the base class is always kept) — a
|
||||||
|
``viewBox``-only SVG has NO intrinsic size, so the default 300×150
|
||||||
|
replaced-element box would sit small inside the full-width
|
||||||
|
``.doc-image`` card — pinned as a node behavioral run of the real
|
||||||
|
``docImageBlock`` (svg → the compound class, ``src`` from
|
||||||
|
``image_url`` never inlined, ``alt`` = summary with the title
|
||||||
|
fallback; png → the class string byte-identical to the base class)
|
||||||
|
plus the ``.doc-image-img--vector`` CSS pins (``width: 100%`` +
|
||||||
|
``height: auto`` + ``object-fit: contain`` under the base rule's
|
||||||
|
70vh clamp, zero colour literals, the base ``.doc-image-img`` rule
|
||||||
|
untouched).
|
||||||
"""
|
"""
|
||||||
from __future__ import annotations
|
from __future__ import annotations
|
||||||
|
|
||||||
@@ -2102,3 +2116,180 @@ def test_doc_prose_css_wraps_in_the_reading_column() -> None:
|
|||||||
raw_rule = css[raw_i : css.find("}", raw_i)]
|
raw_rule = css[raw_i : css.find("}", raw_i)]
|
||||||
assert "white-space: pre" in raw_rule
|
assert "white-space: pre" in raw_rule
|
||||||
assert "overflow-x: auto" in raw_rule
|
assert "overflow-x: auto" in raw_rule
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# Phase 181 (task 04) — the SVG vector preview: docImageBlock's format
|
||||||
|
# gate (base class + the doc-image-img--vector modifier, LOCKED A3:
|
||||||
|
# bytes only ever through <img>) and the .doc-image-img--vector CSS
|
||||||
|
# (a viewBox-only SVG has no intrinsic size — width: 100% spans the
|
||||||
|
# .doc-image card, object-fit: contain survives the 70vh clamp,
|
||||||
|
# palette-neutral). The chat figure (app.js) and the Sources
|
||||||
|
# thumbnail (sources.js) are VERIFIED non-changes (design D7): both
|
||||||
|
# key off image_url alone and their boxes pin the img
|
||||||
|
# (.source-image-img 96px cap, .kb-doc-thumb-img fixed 48px box).
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
def test_doc_image_block_gates_the_vector_modifier_on_format() -> None:
|
||||||
|
"""Source pins (node-free): docImageBlock selects the compound
|
||||||
|
class on ``doc.format === "svg"`` and the bare base class
|
||||||
|
otherwise (every current .doc-image-img rule keeps applying);
|
||||||
|
the bytes stay an <img> src property assignment — never inlined
|
||||||
|
(LOCKED A3), never innerHTML. The comment records the rationale
|
||||||
|
(no-intrinsic-size / 300×150) and the legacy-TEXT honesty
|
||||||
|
(XML-in-.doc-raw is the truth; no content sniffing)."""
|
||||||
|
js = _read(DOCUMENT_JS)
|
||||||
|
fn = _fn(js, "docImageBlock")
|
||||||
|
assert 'doc.format === "svg"' in fn
|
||||||
|
assert '"doc-image-img doc-image-img--vector"' in fn
|
||||||
|
assert ': "doc-image-img";' in fn, "the raster path keeps the bare base class"
|
||||||
|
assert "img.src = doc.image_url;" in fn, "the bytes URL stays a property assignment"
|
||||||
|
assert "innerHTML" not in fn
|
||||||
|
# The rationale + honesty comments (house style: why, not what).
|
||||||
|
for citation in ("Phase 181", "viewBox", "300\u00d7150", "LOCKED A3", ".doc-raw"):
|
||||||
|
assert citation in fn, f"the comment must cite {citation!r}"
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.skipif(not HAVE_NODE, reason="node not available")
|
||||||
|
def test_doc_image_block_svg_yields_the_vector_class_png_unchanged() -> None:
|
||||||
|
"""Behavioral (node, the house fake DOM): the REAL docImageBlock —
|
||||||
|
an SVG doc renders class="doc-image-img doc-image-img--vector"
|
||||||
|
with src = image_url and alt = the summary (title fallback when
|
||||||
|
the summary is null/blank); a PNG doc's class string is the base
|
||||||
|
class ONLY (byte-identical to pre-phase — the phase-122/133
|
||||||
|
raster surfaces cannot drift); the wrapper stays .doc-image."""
|
||||||
|
js = _read(DOCUMENT_JS)
|
||||||
|
# The docs ride in via json.dumps (the summary carries quotes and
|
||||||
|
# an em-dash — the alt contract must survive verbatim).
|
||||||
|
cases = {
|
||||||
|
"svg": {
|
||||||
|
"format": "svg",
|
||||||
|
"image_url": "/api/documents/7/image",
|
||||||
|
"summary": 'Vector diagram "topo" \u2014 3 shapes',
|
||||||
|
"title": "topo",
|
||||||
|
},
|
||||||
|
"svgNoSummary": {
|
||||||
|
"format": "svg",
|
||||||
|
"image_url": "/api/documents/9/image",
|
||||||
|
"summary": None,
|
||||||
|
"title": "rack",
|
||||||
|
},
|
||||||
|
"svgBlankSummary": {
|
||||||
|
"format": "svg",
|
||||||
|
"image_url": "/api/documents/10/image",
|
||||||
|
"summary": " ",
|
||||||
|
"title": "blank",
|
||||||
|
},
|
||||||
|
"png": {
|
||||||
|
"format": "png",
|
||||||
|
"image_url": "/api/documents/8/image",
|
||||||
|
"summary": "A photo",
|
||||||
|
"title": "t",
|
||||||
|
},
|
||||||
|
}
|
||||||
|
script = (
|
||||||
|
_CONTENT_BRANCH_FAKE_DOM
|
||||||
|
+ _fn(js, "docImageBlock")
|
||||||
|
+ "\nconst cases = "
|
||||||
|
+ json.dumps(cases)
|
||||||
|
+ """
|
||||||
|
;
|
||||||
|
function imageFor(doc) {
|
||||||
|
const wrap = docImageBlock(doc);
|
||||||
|
const img = wrap.children[0];
|
||||||
|
return {
|
||||||
|
wrapClass: wrap.className, imgClass: img.className, src: img.src, alt: img.alt,
|
||||||
|
};
|
||||||
|
}
|
||||||
|
const out = {};
|
||||||
|
for (const [k, doc] of Object.entries(cases)) out[k] = imageFor(doc);
|
||||||
|
console.log(JSON.stringify(out));
|
||||||
|
"""
|
||||||
|
)
|
||||||
|
out = json.loads(_run_node(script))
|
||||||
|
assert out["svg"] == {
|
||||||
|
"wrapClass": "doc-image",
|
||||||
|
"imgClass": "doc-image-img doc-image-img--vector",
|
||||||
|
"src": "/api/documents/7/image",
|
||||||
|
"alt": 'Vector diagram "topo" \u2014 3 shapes',
|
||||||
|
}
|
||||||
|
# alt falls back to the title (the WCAG alt contract, unchanged).
|
||||||
|
assert out["svgNoSummary"]["alt"] == "rack"
|
||||||
|
assert out["svgBlankSummary"]["alt"] == "blank"
|
||||||
|
assert out["svgBlankSummary"]["imgClass"] == "doc-image-img doc-image-img--vector"
|
||||||
|
# The raster class string is byte-identical to pre-phase (base only).
|
||||||
|
assert out["png"]["imgClass"] == "doc-image-img"
|
||||||
|
assert out["png"]["src"] == "/api/documents/8/image"
|
||||||
|
|
||||||
|
|
||||||
|
def test_doc_image_vector_css_spans_the_card_without_squashing() -> None:
|
||||||
|
"""styles.css: ``.doc-image-img--vector`` (immediately after
|
||||||
|
``.doc-image-img``) is ``width: 100%`` + ``height: auto`` +
|
||||||
|
``object-fit: contain`` — a viewBox-only SVG (no intrinsic size:
|
||||||
|
the default replaced-element box is 300\u00d7150) spans the
|
||||||
|
full-width .doc-image card, and contain letterboxes instead of
|
||||||
|
squashing when the base rule's max-height: 70vh clamps a tall
|
||||||
|
drawing. Zero colour literals (palette-neutral — the monochrome
|
||||||
|
invariant); the base .doc-image-img rule is UNTOUCHED (the
|
||||||
|
phase-122 raster contract)."""
|
||||||
|
css = _read(STYLES_CSS)
|
||||||
|
base_i = css.find(".doc-image-img {")
|
||||||
|
vec_i = css.find(".doc-image-img--vector {")
|
||||||
|
assert -1 < base_i < vec_i, ".doc-image-img--vector must sit next to the base rule"
|
||||||
|
# Provenance comment directly above the rule (house style).
|
||||||
|
comment_start = css.rfind("/*", 0, vec_i)
|
||||||
|
provenance = css[comment_start:vec_i]
|
||||||
|
assert -1 < comment_start < vec_i
|
||||||
|
for citation in ("phase 181", "viewBox", "300\u00d7150", "object-fit", "doc.format"):
|
||||||
|
assert citation in provenance, f"the comment must cite {citation!r}"
|
||||||
|
assert "no animation" in provenance
|
||||||
|
rule = css[vec_i : css.find("}", vec_i)]
|
||||||
|
for decl in ("width: 100%", "height: auto", "object-fit: contain"):
|
||||||
|
assert decl in rule, f".doc-image-img--vector must carry {decl!r}"
|
||||||
|
# Palette-neutral — zero colour literals (no new hue).
|
||||||
|
assert not re.search(r"#[0-9a-fA-F]{3,8}\b", rule)
|
||||||
|
assert "rgb(" not in rule and "hsl(" not in rule and "color-mix(" not in rule
|
||||||
|
# The untouched base rule (the raster + onerror contract).
|
||||||
|
base_rule = css[base_i : css.find("}", base_i)]
|
||||||
|
for decl in ("max-width: 100%", "max-height: 70vh", "height: auto"):
|
||||||
|
assert decl in base_rule, f"the base rule must keep {decl!r}"
|
||||||
|
|
||||||
|
|
||||||
|
def test_figure_and_thumbnail_key_off_image_url_alone() -> None:
|
||||||
|
"""Design D7's verified non-changes: the chat figure
|
||||||
|
(appendSourceImageFigure) and the Sources KB-tree thumbnail
|
||||||
|
(docThumb) resolve their bytes from ``image_url`` alone — no
|
||||||
|
format field on the wire, none added — and their CSS boxes pin
|
||||||
|
the img (.source-image-img: 96px cap + object-fit: contain; the
|
||||||
|
.kb-doc-thumb-img: the fixed 48px box), so a size-less SVG
|
||||||
|
scales inside each cap exactly like a raster. Pins: neither
|
||||||
|
function sniffs a format at all (the deliberate non-change,
|
||||||
|
design D7), the phase-181 comments sit where they were made,
|
||||||
|
and the two rules keep their box declarations."""
|
||||||
|
app_js = _read(APP_JS)
|
||||||
|
assert app_js.count("function appendSourceImageFigure(") == 1
|
||||||
|
fig = _fn(app_js, "appendSourceImageFigure")
|
||||||
|
assert "img.src = s.image_url;" in fig
|
||||||
|
assert "format" not in fig, "no format sniffing on the chat figure (D7)"
|
||||||
|
fn_start = app_js.index("function appendSourceImageFigure(")
|
||||||
|
assert "phase 181" in app_js[max(0, fn_start - 2400) : fn_start].lower(), (
|
||||||
|
"the deliberate-non-change comment sits above the figure"
|
||||||
|
)
|
||||||
|
sources_js = _read(SOURCES_JS)
|
||||||
|
thumb_start = sources_js.index("function docThumb(d)")
|
||||||
|
thumb_end = sources_js.index("\n }", thumb_start)
|
||||||
|
thumb = sources_js[thumb_start:thumb_end]
|
||||||
|
assert "img.src = d.image_url;" in thumb
|
||||||
|
assert "format" not in thumb, "no format sniffing on the thumbnail (D7)"
|
||||||
|
branch_i = sources_js.index("if (d.is_image) {")
|
||||||
|
assert "phase 181" in sources_js[branch_i - 900 : branch_i].lower(), (
|
||||||
|
"the verified-non-change comment sits above the is_image branch"
|
||||||
|
)
|
||||||
|
css = _read(STYLES_CSS)
|
||||||
|
cap_i = css.find(".source-image-img {")
|
||||||
|
cap = css[cap_i : css.find("}", cap_i)]
|
||||||
|
assert "max-height: 96px" in cap and "object-fit: contain" in cap
|
||||||
|
box_i = css.find(".kb-doc-thumb-img {")
|
||||||
|
box = css[box_i : css.find("}", box_i)]
|
||||||
|
assert "width: 100%" in box and "height: 100%" in box
|
||||||
|
|||||||
@@ -106,19 +106,25 @@ def _settings(**kwargs: Any) -> Settings:
|
|||||||
|
|
||||||
|
|
||||||
def test_image_defaults_match_locked_a3() -> None:
|
def test_image_defaults_match_locked_a3() -> None:
|
||||||
"""LOCKED A3: image support is OFF by default (enable only when the
|
"""LOCKED A3 + phase 181 LOCKED A7: image support is OFF by default
|
||||||
chat model supports vision), the built-in extension set is the six
|
(enable only when the chat model supports vision), the built-in
|
||||||
common formats, and the image bytes' persistent home is the
|
extension set is the six common raster formats PLUS ``svg`` (the one
|
||||||
|
VECTOR member — A1: it indexes from its own extracted text, never a
|
||||||
|
vision call), and the image bytes' persistent home is the
|
||||||
``~/bor-sources/images`` raw string (``expanduser`` is the
|
``~/bor-sources/images`` raw string (``expanduser`` is the
|
||||||
importer's job — the ``sources_dir``/``upload_dir`` convention)."""
|
importer's job — the ``sources_dir``/``upload_dir`` convention).
|
||||||
|
``images`` stays the SINGLE knob (A7): with it off an ``.svg`` is a
|
||||||
|
plain text document, no special-casing."""
|
||||||
s = _settings()
|
s = _settings()
|
||||||
assert s.images is False
|
assert s.images is False
|
||||||
# Dotted, lowercased form — the shape the walk's matcher consumes
|
# Dotted, lowercased form — the shape the walk's matcher consumes
|
||||||
# (the ``import_extension_set`` precedent).
|
# (the ``import_extension_set`` precedent).
|
||||||
assert s.image_extension_set == {".png", ".jpg", ".jpeg", ".webp", ".gif", ".bmp"}
|
assert s.image_extension_set == {
|
||||||
|
".png", ".jpg", ".jpeg", ".webp", ".gif", ".bmp", ".svg",
|
||||||
|
}
|
||||||
assert s.image_dir == "~/bor-sources/images"
|
assert s.image_dir == "~/bor-sources/images"
|
||||||
# The raw string is stored untouched (no expanduser here).
|
# The raw string is stored untouched (no expanduser here).
|
||||||
assert s.image_extensions == "png,jpg,jpeg,webp,gif,bmp"
|
assert s.image_extensions == "png,jpg,jpeg,webp,gif,bmp,svg"
|
||||||
|
|
||||||
|
|
||||||
def test_image_set_is_separate_from_the_import_set() -> None:
|
def test_image_set_is_separate_from_the_import_set() -> None:
|
||||||
@@ -127,7 +133,9 @@ def test_image_set_is_separate_from_the_import_set() -> None:
|
|||||||
the import set leaves the image set untouched, and vice versa."""
|
the import set leaves the image set untouched, and vice versa."""
|
||||||
s = _settings(import_extensions="md")
|
s = _settings(import_extensions="md")
|
||||||
assert s.import_extension_set == {".md"}
|
assert s.import_extension_set == {".md"}
|
||||||
assert s.image_extension_set == {".png", ".jpg", ".jpeg", ".webp", ".gif", ".bmp"}
|
assert s.image_extension_set == {
|
||||||
|
".png", ".jpg", ".jpeg", ".webp", ".gif", ".bmp", ".svg",
|
||||||
|
}
|
||||||
s2 = _settings(image_extensions="md")
|
s2 = _settings(image_extensions="md")
|
||||||
assert s2.image_extension_set == {".md"}
|
assert s2.image_extension_set == {".md"}
|
||||||
assert ".md" in s2.import_extension_set # the default import set is intact
|
assert ".md" in s2.import_extension_set # the default import set is intact
|
||||||
@@ -762,7 +770,16 @@ def test_image_mimes_cover_the_default_family() -> None:
|
|||||||
default image family — dotted lowercase keys, the
|
default image family — dotted lowercase keys, the
|
||||||
``image_extension_set`` shape — and the unlisted-token fallback is
|
``image_extension_set`` shape — and the unlisted-token fallback is
|
||||||
the generic mime (a custom ``BOR_IMAGE_EXTENSIONS`` format fails
|
the generic mime (a custom ``BOR_IMAGE_EXTENSIONS`` format fails
|
||||||
soft at the vision endpoint, never a guessed type)."""
|
soft at the vision endpoint, never a guessed type).
|
||||||
|
|
||||||
|
The map is pinned by IDENTITY (not membership), so a silent removal
|
||||||
|
or re-pointing of any entry fails here. Phase 181 (task 03) adds
|
||||||
|
``.svg`` as the ONE vector entry; the six raster entries are
|
||||||
|
untouched by it, and the vector import branch never reaches
|
||||||
|
:func:`image_data_url` (LOCKED A1: no vision call for an SVG), so
|
||||||
|
the addition cannot change a describe call — the value exists for
|
||||||
|
the BYTES route (``app.api.docs.get_document_image`` reads the same
|
||||||
|
map for its ``Content-Type`` and its sandboxed-CSP guard)."""
|
||||||
assert IMAGE_MIMES == {
|
assert IMAGE_MIMES == {
|
||||||
".png": "image/png",
|
".png": "image/png",
|
||||||
".jpg": "image/jpeg",
|
".jpg": "image/jpeg",
|
||||||
@@ -770,8 +787,12 @@ def test_image_mimes_cover_the_default_family() -> None:
|
|||||||
".webp": "image/webp",
|
".webp": "image/webp",
|
||||||
".gif": "image/gif",
|
".gif": "image/gif",
|
||||||
".bmp": "image/bmp",
|
".bmp": "image/bmp",
|
||||||
|
".svg": "image/svg+xml",
|
||||||
}
|
}
|
||||||
assert IMAGE_FALLBACK_MIME == "application/octet-stream"
|
assert IMAGE_FALLBACK_MIME == "application/octet-stream"
|
||||||
|
# Every member of the default image set now has a mime — raster and
|
||||||
|
# vector alike (the vector one is serve-side only, per above).
|
||||||
|
assert ".svg" in _settings().image_extension_set
|
||||||
assert all(ext in IMAGE_MIMES for ext in _settings().image_extension_set)
|
assert all(ext in IMAGE_MIMES for ext in _settings().image_extension_set)
|
||||||
|
|
||||||
|
|
||||||
|
|||||||
@@ -1008,6 +1008,7 @@ def test_import_summary_log_line_includes_summary_counters(
|
|||||||
s.summary_backfilled = 1
|
s.summary_backfilled = 1
|
||||||
s.dates_updated = 0
|
s.dates_updated = 0
|
||||||
s.images_failed = 0
|
s.images_failed = 0
|
||||||
|
s.svg_indexed = 4
|
||||||
s.image_context_backfilled = 2
|
s.image_context_backfilled = 2
|
||||||
s.pdfs_screenshotted = 1
|
s.pdfs_screenshotted = 1
|
||||||
s.pdfs_pictorial = 2
|
s.pdfs_pictorial = 2
|
||||||
@@ -1021,7 +1022,7 @@ def test_import_summary_log_line_includes_summary_counters(
|
|||||||
assert line == (
|
assert line == (
|
||||||
"import: summary files=3 added=3 updated=0 unchanged=0 pruned=0 errors=0 "
|
"import: summary files=3 added=3 updated=0 unchanged=0 pruned=0 errors=0 "
|
||||||
"chunks=5 embed_batches=4 summaries=2 summary_errors=1 summary_backfilled=1 "
|
"chunks=5 embed_batches=4 summaries=2 summary_errors=1 summary_backfilled=1 "
|
||||||
"dates_updated=0 images_failed=0 image_context_backfilled=2 "
|
"dates_updated=0 images_failed=0 svg_indexed=4 image_context_backfilled=2 "
|
||||||
"pdfs_screenshotted=1 pdfs_pictorial=2 pdf_pictorial_pages=5 "
|
"pdfs_screenshotted=1 pdfs_pictorial=2 pdf_pictorial_pages=5 "
|
||||||
"pdf_images_deduped=3 pdf_images_blank_skipped=1 "
|
"pdf_images_deduped=3 pdf_images_blank_skipped=1 "
|
||||||
"formats=yaml:2,md:1"
|
"formats=yaml:2,md:1"
|
||||||
|
|||||||
@@ -0,0 +1,482 @@
|
|||||||
|
"""Unit: phase 181 task 02 — an ``.svg`` indexes as a VECTOR image document.
|
||||||
|
|
||||||
|
The pure extractor (:mod:`app.rag.svg_text`) is pinned in
|
||||||
|
``tests/unit/test_svg_text_extraction.py``; this module pins what the
|
||||||
|
IMPORTER does with it (LOCKED A1/A2/A7):
|
||||||
|
|
||||||
|
* the vector branch of :func:`app.rag.importer._index_image_file` — an
|
||||||
|
``.svg`` lands ``is_image=True`` with ``image_path`` under
|
||||||
|
``image_dir``, its ``content`` is the file's OWN extracted text, its
|
||||||
|
``summary`` mirrors it (the phase-122 convention), and the vision seam
|
||||||
|
:func:`app.rag.importer._describe_or_skip` is NEVER called (the seam is
|
||||||
|
patched with a recording stand-in that would return a sentinel — the
|
||||||
|
empty recording list + the content assertion are the proof, and a
|
||||||
|
call-counting seam cannot silently succeed);
|
||||||
|
* the log-only ``svg_indexed`` counter and the ``import: svg indexed``
|
||||||
|
auditability line (AGENTS rule 10), including ``structural=True`` for
|
||||||
|
a drawing whose only words are the LOCKED A2 fallback line;
|
||||||
|
* the fail-soft (LOCKED A2's boundary): a file the extractor REFUSES
|
||||||
|
(over the byte cap, malformed XML) skips exactly like an undescribed
|
||||||
|
raster — ``images_failed``, a warning, no row, no orphan copy — while
|
||||||
|
a merely textless SVG does NOT skip;
|
||||||
|
* the phase-125 ``image_context`` enrichment prepends for an SVG exactly
|
||||||
|
as for a raster;
|
||||||
|
* the phase-181 fast-path kind guard (:func:`_row_kind_mismatch`): a row
|
||||||
|
imported as TEXT before ``svg`` joined ``BOR_IMAGE_EXTENSIONS`` is
|
||||||
|
converted on the NEXT sync (same bytes, same ``file_hash`` — and for an
|
||||||
|
ASCII file the text digest equals the byte digest too, which is why the
|
||||||
|
image branch's own content-digest skip now requires ``is_image``), and
|
||||||
|
a row already of the right kind — raster or vector — keeps the
|
||||||
|
phase-147 fast path byte-for-byte (no second seam call, no second
|
||||||
|
``svg_indexed``).
|
||||||
|
"""
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import asyncio
|
||||||
|
import hashlib
|
||||||
|
import logging
|
||||||
|
from pathlib import Path
|
||||||
|
from typing import Any
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
from sqlalchemy import select
|
||||||
|
|
||||||
|
import app.rag.importer as importer
|
||||||
|
from app.config import Settings
|
||||||
|
from app.models import Chunk, Document
|
||||||
|
from app.rag.importer import import_sources
|
||||||
|
from app.rag.svg_text import MAX_SVG_BYTES
|
||||||
|
from tests.fakes import FakeEmbedder
|
||||||
|
|
||||||
|
#: A diagram that carries words: ``<title>`` + ``<desc>`` + a ``<text>``
|
||||||
|
#: inside a group (document order is the extractor's contract).
|
||||||
|
SVG_TOPO = (
|
||||||
|
b'<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 120 80">'
|
||||||
|
b"<title>Server rack topology</title>"
|
||||||
|
b"<desc>Two switches uplink to the router.</desc>"
|
||||||
|
b'<g><path d="M0 0h10"/><text>rack-1</text></g>'
|
||||||
|
b"</svg>"
|
||||||
|
)
|
||||||
|
|
||||||
|
#: The text :func:`app.rag.svg_text.extract_svg_text` yields for
|
||||||
|
#: :data:`SVG_TOPO` (pinned literally here so a change to the extraction
|
||||||
|
#: contract breaks BOTH modules, not silently).
|
||||||
|
SVG_TOPO_TEXT = "Server rack topology\nTwo switches uplink to the router.\nrack-1"
|
||||||
|
|
||||||
|
#: A pure-shape drawing: no words at all (the LOCKED A2 class).
|
||||||
|
SVG_SHAPES_ONLY = (
|
||||||
|
b'<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 64 64">'
|
||||||
|
b'<path d="M0 0h10"/><path d="M0 20h10"/><circle cx="1" cy="1" r="2"/>'
|
||||||
|
b"</svg>"
|
||||||
|
)
|
||||||
|
|
||||||
|
#: The LOCKED A2 structural line for :data:`SVG_SHAPES_ONLY` — built only
|
||||||
|
#: from facts the file states (stem, shape census, viewBox).
|
||||||
|
SVG_SHAPES_ONLY_TEXT = (
|
||||||
|
'Vector diagram "gizmo" — 3 shapes (path×2, circle), viewBox 64×64'
|
||||||
|
)
|
||||||
|
|
||||||
|
#: Over :data:`MAX_SVG_BYTES`: the size guard refuses it BEFORE any parse.
|
||||||
|
SVG_OVER_CAP = b"<svg>" + b"\n" * (MAX_SVG_BYTES + 1) + b"</svg>"
|
||||||
|
|
||||||
|
#: Well-formed bytes, unparseable markup (the second refusal class).
|
||||||
|
SVG_MALFORMED = b'<svg xmlns="http://www.w3.org/2000/svg"><title>oops'
|
||||||
|
|
||||||
|
#: A real 1×1 PNG for the raster fast-path regression pins.
|
||||||
|
PNG_1X1 = b"\x89PNG\r\n\x1a\n" + b"\x00" * 24
|
||||||
|
|
||||||
|
#: What a describe seam would hand back if the vector branch ever reached
|
||||||
|
#: it — never a plausible diagram text, so it cannot pass unnoticed.
|
||||||
|
VISION_SENTINEL = "SHOULD NEVER BE DESCRIBED: a vision call ran for an SVG."
|
||||||
|
|
||||||
|
|
||||||
|
def _settings(**kwargs: Any) -> Settings:
|
||||||
|
"""Build Settings without reading a .env file (deterministic tests)."""
|
||||||
|
kwargs.setdefault("_env_file", None)
|
||||||
|
return Settings(**kwargs) # pyright: ignore[reportCallIssue] (kwarg exists at runtime)
|
||||||
|
|
||||||
|
|
||||||
|
def _image_llm(tmp_path: Path, **kwargs: Any) -> FakeEmbedder:
|
||||||
|
"""A fake LLM whose settings index images into *tmp_path*."""
|
||||||
|
kwargs.setdefault("_env_file", None)
|
||||||
|
kwargs.setdefault("images", True)
|
||||||
|
kwargs.setdefault("image_dir", str(tmp_path / "images"))
|
||||||
|
llm = FakeEmbedder()
|
||||||
|
llm.settings = Settings(**kwargs) # pyright: ignore[reportCallIssue]
|
||||||
|
return llm
|
||||||
|
|
||||||
|
|
||||||
|
def _record_description(monkeypatch: pytest.MonkeyPatch) -> list[str]:
|
||||||
|
"""Patch the vision seam with a recorder (never a real call): the
|
||||||
|
paths it is asked to describe, always answered with
|
||||||
|
:data:`VISION_SENTINEL`. An empty list + a non-sentinel ``content``
|
||||||
|
is the proof that the vector branch bypassed vision entirely."""
|
||||||
|
handed: list[str] = []
|
||||||
|
|
||||||
|
async def _fake(
|
||||||
|
llm: importer.Embedder,
|
||||||
|
*,
|
||||||
|
data: bytes,
|
||||||
|
source: str,
|
||||||
|
rel: str,
|
||||||
|
full_path: Path,
|
||||||
|
context: str | None = None,
|
||||||
|
) -> str | None:
|
||||||
|
handed.append(rel)
|
||||||
|
return VISION_SENTINEL
|
||||||
|
|
||||||
|
monkeypatch.setattr(importer, "_describe_or_skip", _fake)
|
||||||
|
return handed
|
||||||
|
|
||||||
|
|
||||||
|
def _doc(db, source: str, rel: str) -> Document:
|
||||||
|
doc = db.scalar(
|
||||||
|
select(Document).where(Document.source == source, Document.path == rel)
|
||||||
|
)
|
||||||
|
assert doc is not None, f"no documents row for ({source!r}, {rel!r})"
|
||||||
|
return doc
|
||||||
|
|
||||||
|
|
||||||
|
def _cleanup_source(db, source: str) -> None:
|
||||||
|
for doc in db.scalars(select(Document).where(Document.source == source)).all():
|
||||||
|
db.delete(doc)
|
||||||
|
db.commit()
|
||||||
|
|
||||||
|
|
||||||
|
# --- the vector branch ------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
def test_svg_indexes_as_an_image_document_with_no_vision_call(
|
||||||
|
db, tmp_path: Path, monkeypatch: pytest.MonkeyPatch, caplog: pytest.LogCaptureFixture
|
||||||
|
) -> None:
|
||||||
|
"""LOCKED A1: the ``.svg`` takes the image branch (``is_image``,
|
||||||
|
``image_path``, the ``<doc-id>.svg`` copy, the ``summary = content``
|
||||||
|
convention, one embedded content chunk + the ``is_summary`` chunk)
|
||||||
|
and its searchable text is the file's own ``<title>``/``<desc>``/
|
||||||
|
``<text>`` — with the vision seam never invoked. ``svg_indexed`` and
|
||||||
|
the per-file ``import: svg indexed`` line make the run auditable."""
|
||||||
|
root = tmp_path / "svgsrc"
|
||||||
|
root.mkdir()
|
||||||
|
(root / "topo.svg").write_bytes(SVG_TOPO)
|
||||||
|
llm = _image_llm(tmp_path)
|
||||||
|
handed = _record_description(monkeypatch)
|
||||||
|
|
||||||
|
try:
|
||||||
|
with caplog.at_level(logging.INFO, logger="app.importer"):
|
||||||
|
summary = asyncio.run(import_sources([root], llm, session=db))
|
||||||
|
assert (summary.files, summary.added, summary.images_failed) == (1, 1, 0)
|
||||||
|
assert summary.errors == 0
|
||||||
|
assert summary.formats == {"svg": 1} # counted under its image token
|
||||||
|
assert summary.svg_indexed == 1
|
||||||
|
assert handed == [], "an SVG must never reach the vision seam"
|
||||||
|
|
||||||
|
doc = _doc(db, root.name, "topo.svg")
|
||||||
|
assert doc.is_image is True
|
||||||
|
assert doc.title == "topo" # the non-markdown stem rule (shared with raster)
|
||||||
|
assert doc.content == SVG_TOPO_TEXT
|
||||||
|
assert doc.summary == doc.content # the phase-122 image convention
|
||||||
|
assert doc.image_context is None # standalone: the pre-phase row shape
|
||||||
|
assert doc.image_path is not None
|
||||||
|
copy = Path(doc.image_path)
|
||||||
|
assert copy.parent == Path(llm.settings.image_dir).expanduser()
|
||||||
|
assert copy.name == f"{doc.id}.svg" # row and copy agree on the id
|
||||||
|
assert copy.read_bytes() == SVG_TOPO # the exact bytes, unserved here
|
||||||
|
|
||||||
|
chunks = db.scalars(select(Chunk).where(Chunk.document_id == doc.id)).all()
|
||||||
|
content_chunks = [c for c in chunks if not c.is_summary]
|
||||||
|
assert [c.content for c in content_chunks] == [doc.content]
|
||||||
|
summary_chunks = [c for c in chunks if c.is_summary]
|
||||||
|
assert len(summary_chunks) == 1 and summary_chunks[0].position == -1
|
||||||
|
assert summary_chunks[0].content == doc.content
|
||||||
|
for c in chunks:
|
||||||
|
assert c.embedding is not None and len(c.embedding) == 768
|
||||||
|
|
||||||
|
indexed = [
|
||||||
|
r.getMessage() for r in caplog.records if "import: svg indexed" in r.getMessage()
|
||||||
|
]
|
||||||
|
assert indexed == [
|
||||||
|
f"import: svg indexed source={root.name} path=topo.svg "
|
||||||
|
f"structural=False chars={len(SVG_TOPO_TEXT)}"
|
||||||
|
]
|
||||||
|
line = next(
|
||||||
|
r.getMessage()
|
||||||
|
for r in caplog.records
|
||||||
|
if r.getMessage().startswith("import: summary files=")
|
||||||
|
)
|
||||||
|
assert "svg_indexed=1" in line and "images_failed=0" in line
|
||||||
|
finally:
|
||||||
|
_cleanup_source(db, root.name)
|
||||||
|
|
||||||
|
|
||||||
|
def test_textless_svg_indexes_with_the_structural_line(
|
||||||
|
db, tmp_path: Path, monkeypatch: pytest.MonkeyPatch, caplog: pytest.LogCaptureFixture
|
||||||
|
) -> None:
|
||||||
|
"""LOCKED A2: a diagram of pure ``<path>``s is NOT skipped — it is
|
||||||
|
indexed with the structural line built from the file (stem, shape
|
||||||
|
census, viewBox), counted like any other vector doc and logged with
|
||||||
|
``structural=True`` so the owner can tell it from a worded diagram."""
|
||||||
|
root = tmp_path / "shapeonly"
|
||||||
|
root.mkdir()
|
||||||
|
(root / "gizmo.svg").write_bytes(SVG_SHAPES_ONLY)
|
||||||
|
llm = _image_llm(tmp_path)
|
||||||
|
handed = _record_description(monkeypatch)
|
||||||
|
try:
|
||||||
|
with caplog.at_level(logging.INFO, logger="app.importer"):
|
||||||
|
summary = asyncio.run(import_sources([root], llm, session=db))
|
||||||
|
assert (summary.files, summary.added, summary.images_failed) == (1, 1, 0)
|
||||||
|
assert summary.svg_indexed == 1
|
||||||
|
assert handed == []
|
||||||
|
doc = _doc(db, root.name, "gizmo.svg")
|
||||||
|
assert doc.is_image is True
|
||||||
|
assert doc.content == SVG_SHAPES_ONLY_TEXT
|
||||||
|
assert doc.summary == doc.content
|
||||||
|
line = next(
|
||||||
|
r.getMessage() for r in caplog.records if "import: svg indexed" in r.getMessage()
|
||||||
|
)
|
||||||
|
assert "structural=True" in line
|
||||||
|
finally:
|
||||||
|
_cleanup_source(db, root.name)
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.parametrize(
|
||||||
|
("name", "payload"),
|
||||||
|
[
|
||||||
|
pytest.param("huge.svg", SVG_OVER_CAP, id="over-the-byte-cap"),
|
||||||
|
pytest.param("broken.svg", SVG_MALFORMED, id="malformed-xml"),
|
||||||
|
],
|
||||||
|
)
|
||||||
|
def test_unusable_svg_skips_the_doc_counts_and_logs(
|
||||||
|
db,
|
||||||
|
tmp_path: Path,
|
||||||
|
monkeypatch: pytest.MonkeyPatch,
|
||||||
|
caplog: pytest.LogCaptureFixture,
|
||||||
|
name: str,
|
||||||
|
payload: bytes,
|
||||||
|
) -> None:
|
||||||
|
"""The fail-soft stays LOCKED A3-shaped for the vector case: a file
|
||||||
|
the extractor refuses skips the doc ENTIRELY — no row, no copy (not
|
||||||
|
even the image dir), ``images_failed`` + a warning naming the vector
|
||||||
|
extraction, the sync continues. This is the ONLY skip class the phase
|
||||||
|
keeps for an SVG (:func:`app.rag.svg_text.extract_svg_text`)."""
|
||||||
|
root = tmp_path / "badsvg"
|
||||||
|
root.mkdir()
|
||||||
|
(root / name).write_bytes(payload)
|
||||||
|
llm = _image_llm(tmp_path)
|
||||||
|
handed = _record_description(monkeypatch)
|
||||||
|
try:
|
||||||
|
with caplog.at_level(logging.INFO, logger="app.importer"):
|
||||||
|
summary = asyncio.run(import_sources([root], llm, session=db))
|
||||||
|
assert (summary.files, summary.added, summary.images_failed) == (1, 0, 1)
|
||||||
|
assert summary.svg_indexed == 0
|
||||||
|
assert handed == []
|
||||||
|
assert db.scalar(select(Document).where(Document.source == root.name)) is None
|
||||||
|
assert not Path(llm.settings.image_dir).expanduser().exists(), (
|
||||||
|
"no copy — and not even the image dir — may be left behind"
|
||||||
|
)
|
||||||
|
warnings = [
|
||||||
|
r
|
||||||
|
for r in caplog.records
|
||||||
|
if "svg text extraction failed" in r.getMessage()
|
||||||
|
]
|
||||||
|
assert len(warnings) == 1 and warnings[0].levelno == logging.WARNING
|
||||||
|
assert "image description failed" not in caplog.text
|
||||||
|
line = next(
|
||||||
|
r.getMessage()
|
||||||
|
for r in caplog.records
|
||||||
|
if r.getMessage().startswith("import: summary files=")
|
||||||
|
)
|
||||||
|
assert "images_failed=1" in line and "svg_indexed=0" in line
|
||||||
|
finally:
|
||||||
|
_cleanup_source(db, root.name)
|
||||||
|
|
||||||
|
|
||||||
|
def test_markdown_reference_context_prepends_for_an_svg(
|
||||||
|
db, tmp_path: Path, monkeypatch: pytest.MonkeyPatch
|
||||||
|
) -> None:
|
||||||
|
"""Phase 125's enrichment is kind-agnostic: a ````
|
||||||
|
reference in the same source still builds ``image_context`` (from the
|
||||||
|
file on disk, no vision call) and the doc's content becomes
|
||||||
|
``<context>\\n\\n<extracted text>`` — context first for retrieval, the
|
||||||
|
vector file's own words second."""
|
||||||
|
root = tmp_path / "svganchor"
|
||||||
|
root.mkdir()
|
||||||
|
(root / "guide.md").write_text(
|
||||||
|
"Why this diagram exists.\n\n\n\nWhat follows it.\n",
|
||||||
|
encoding="utf-8",
|
||||||
|
)
|
||||||
|
(root / "topo.svg").write_bytes(SVG_TOPO)
|
||||||
|
llm = _image_llm(tmp_path)
|
||||||
|
handed = _record_description(monkeypatch)
|
||||||
|
try:
|
||||||
|
summary = asyncio.run(import_sources([root], llm, session=db))
|
||||||
|
assert (summary.files, summary.added, summary.images_failed) == (2, 2, 0)
|
||||||
|
assert summary.svg_indexed == 1
|
||||||
|
assert handed == []
|
||||||
|
doc = _doc(db, root.name, "topo.svg")
|
||||||
|
assert doc.is_image is True
|
||||||
|
assert doc.image_context is not None
|
||||||
|
assert doc.image_context.startswith(f"Referenced in {root.name}/guide.md:")
|
||||||
|
assert 'Alt text: "The rack plan"' in doc.image_context
|
||||||
|
assert doc.content == f"{doc.image_context}\n\n{SVG_TOPO_TEXT}"
|
||||||
|
assert doc.summary == doc.content
|
||||||
|
finally:
|
||||||
|
_cleanup_source(db, root.name)
|
||||||
|
|
||||||
|
|
||||||
|
# --- the fast-path kind guard (the text → image migration) ------------------
|
||||||
|
|
||||||
|
|
||||||
|
def test_text_indexed_svg_row_is_converted_on_the_next_sync(
|
||||||
|
db, tmp_path: Path, monkeypatch: pytest.MonkeyPatch
|
||||||
|
) -> None:
|
||||||
|
"""Design D4: the row an earlier run indexed as TEXT (``svg`` not yet
|
||||||
|
in the image set) is converted by the NEXT sync — same bytes, same
|
||||||
|
``file_hash`` — and the sync AFTER that is fast again.
|
||||||
|
|
||||||
|
The conversion needs BOTH halves of the guard: the fast path in
|
||||||
|
:func:`_index_file` must fall through, and the image branch's own
|
||||||
|
content-digest skip must require ``is_image`` — an ASCII file's text
|
||||||
|
digest equals its byte digest, so without it the row would report
|
||||||
|
``unchanged`` forever and never gain a copy (asserted below, because
|
||||||
|
it is the whole reason the guard exists)."""
|
||||||
|
root = tmp_path / "migrate"
|
||||||
|
root.mkdir()
|
||||||
|
(root / "topo.svg").write_bytes(SVG_TOPO)
|
||||||
|
handed = _record_description(monkeypatch)
|
||||||
|
|
||||||
|
# Stage A — the pre-phase state: images OFF, ``svg`` imported as text.
|
||||||
|
text_llm = _image_llm(
|
||||||
|
tmp_path,
|
||||||
|
images=False,
|
||||||
|
import_extensions=f"{_settings().import_extensions},svg",
|
||||||
|
)
|
||||||
|
try:
|
||||||
|
summary_a = asyncio.run(import_sources([root], text_llm, session=db))
|
||||||
|
assert (summary_a.files, summary_a.added, summary_a.svg_indexed) == (1, 1, 0)
|
||||||
|
before = _doc(db, root.name, "topo.svg")
|
||||||
|
assert before.is_image is False
|
||||||
|
assert before.image_path is None
|
||||||
|
assert before.content == SVG_TOPO.decode("utf-8") # the raw XML the owner saw
|
||||||
|
byte_digest = hashlib.sha256(SVG_TOPO).hexdigest()
|
||||||
|
assert before.file_hash == byte_digest
|
||||||
|
# The trap D4 closes: the text digest of a byte-clean file IS the
|
||||||
|
# byte digest, so no digest comparison can spot the mismatch.
|
||||||
|
assert before.content_hash == byte_digest
|
||||||
|
doc_id = before.id
|
||||||
|
|
||||||
|
# Stage B — the toggle on: the guard fires, the row converts.
|
||||||
|
image_llm = _image_llm(tmp_path)
|
||||||
|
summary_b = asyncio.run(import_sources([root], image_llm, session=db))
|
||||||
|
assert (summary_b.added, summary_b.updated, summary_b.unchanged) == (0, 1, 0)
|
||||||
|
assert (summary_b.images_failed, summary_b.svg_indexed) == (0, 1)
|
||||||
|
assert handed == [], "the conversion must not call vision either"
|
||||||
|
after = _doc(db, root.name, "topo.svg")
|
||||||
|
assert after.id == doc_id # converted in place — the id (and copy name) is stable
|
||||||
|
assert after.is_image is True
|
||||||
|
assert after.content == SVG_TOPO_TEXT
|
||||||
|
assert after.summary == after.content
|
||||||
|
assert after.file_hash == byte_digest # the bytes never changed
|
||||||
|
# The image branch stamps the BYTES digest as ``content_hash`` —
|
||||||
|
# the SAME value the text path had already stored, which is why
|
||||||
|
# that skip needed its own ``is_image`` guard.
|
||||||
|
assert after.content_hash == byte_digest
|
||||||
|
copy = Path(str(after.image_path))
|
||||||
|
assert copy == Path(image_llm.settings.image_dir).expanduser() / f"{doc_id}.svg"
|
||||||
|
assert copy.read_bytes() == SVG_TOPO
|
||||||
|
# The text chunks are replaced, not appended: one content chunk
|
||||||
|
# (the extracted text) + the is_summary mirror.
|
||||||
|
chunks = db.scalars(select(Chunk).where(Chunk.document_id == doc_id)).all()
|
||||||
|
assert [c.content for c in chunks if not c.is_summary] == [SVG_TOPO_TEXT]
|
||||||
|
assert [c.content for c in chunks if c.is_summary] == [SVG_TOPO_TEXT]
|
||||||
|
|
||||||
|
# Stage C — the guard fired once: the fast path resumes.
|
||||||
|
summary_c = asyncio.run(import_sources([root], image_llm, session=db))
|
||||||
|
assert (summary_c.added, summary_c.updated, summary_c.unchanged) == (0, 0, 1)
|
||||||
|
assert (summary_c.svg_indexed, summary_c.images_failed) == (0, 0)
|
||||||
|
assert handed == []
|
||||||
|
assert _doc(db, root.name, "topo.svg").is_image is True
|
||||||
|
finally:
|
||||||
|
_cleanup_source(db, root.name)
|
||||||
|
|
||||||
|
|
||||||
|
def test_correctly_classified_raster_row_still_takes_the_fast_path(
|
||||||
|
db, tmp_path: Path, monkeypatch: pytest.MonkeyPatch
|
||||||
|
) -> None:
|
||||||
|
"""The phase-147 fast path survives the guard for every row whose
|
||||||
|
KIND already agrees: a raster image with a matching ``file_hash`` is
|
||||||
|
UNCHANGED on the next sync — the seam is not called a second time and
|
||||||
|
no vector counter moves."""
|
||||||
|
root = tmp_path / "rasterfast"
|
||||||
|
root.mkdir()
|
||||||
|
(root / "pic.png").write_bytes(PNG_1X1)
|
||||||
|
llm = _image_llm(tmp_path)
|
||||||
|
handed = _record_description(monkeypatch)
|
||||||
|
try:
|
||||||
|
first = asyncio.run(import_sources([root], llm, session=db))
|
||||||
|
assert (first.added, first.unchanged, first.svg_indexed) == (1, 0, 0)
|
||||||
|
assert handed == ["pic.png"] # the raster branch DID use the seam
|
||||||
|
doc = _doc(db, root.name, "pic.png")
|
||||||
|
assert doc.is_image is True
|
||||||
|
|
||||||
|
second = asyncio.run(import_sources([root], llm, session=db))
|
||||||
|
assert (second.added, second.updated, second.unchanged) == (0, 0, 1)
|
||||||
|
assert second.svg_indexed == 0
|
||||||
|
assert handed == ["pic.png"], "the fast path must not re-describe the image"
|
||||||
|
assert _doc(db, root.name, "pic.png").content == VISION_SENTINEL
|
||||||
|
finally:
|
||||||
|
_cleanup_source(db, root.name)
|
||||||
|
|
||||||
|
|
||||||
|
def test_correctly_classified_vector_row_still_takes_the_fast_path(
|
||||||
|
db, tmp_path: Path, monkeypatch: pytest.MonkeyPatch
|
||||||
|
) -> None:
|
||||||
|
"""Same pin for the vector row: once converted (or freshly indexed)
|
||||||
|
it is UNCHANGED — no second extraction, no second ``svg_indexed``,
|
||||||
|
no second copy write."""
|
||||||
|
root = tmp_path / "vectorfast"
|
||||||
|
root.mkdir()
|
||||||
|
(root / "topo.svg").write_bytes(SVG_TOPO)
|
||||||
|
llm = _image_llm(tmp_path)
|
||||||
|
handed = _record_description(monkeypatch)
|
||||||
|
try:
|
||||||
|
first = asyncio.run(import_sources([root], llm, session=db))
|
||||||
|
assert (first.added, first.svg_indexed) == (1, 1)
|
||||||
|
copy = Path(str(_doc(db, root.name, "topo.svg").image_path))
|
||||||
|
copy.unlink() # the fast path must not re-create it either
|
||||||
|
|
||||||
|
second = asyncio.run(import_sources([root], llm, session=db))
|
||||||
|
assert (second.added, second.updated, second.unchanged) == (0, 0, 1)
|
||||||
|
assert (second.svg_indexed, second.images_failed) == (0, 0)
|
||||||
|
assert handed == []
|
||||||
|
assert not copy.exists()
|
||||||
|
finally:
|
||||||
|
_cleanup_source(db, root.name)
|
||||||
|
|
||||||
|
|
||||||
|
def test_row_kind_mismatch_table() -> None:
|
||||||
|
"""The guard's whole truth table (pure — no DB, no filesystem):
|
||||||
|
it fires ONLY for a TEXT row whose file the CURRENT settings would
|
||||||
|
index as an image. The reverse direction (an ``is_image`` row while
|
||||||
|
the toggle is off, or a plain text file) stays on the fast path —
|
||||||
|
the phase-122 prune guard owns the off state (LOCKED A7)."""
|
||||||
|
svg = Path("/src/diagrams/topo.svg")
|
||||||
|
md = Path("/src/notes.md")
|
||||||
|
text_row = Document(source="s", path="topo.svg", is_image=False)
|
||||||
|
image_row = Document(source="s", path="topo.svg", is_image=True)
|
||||||
|
|
||||||
|
def _settings_(*, images: bool) -> Settings:
|
||||||
|
return _settings(images=images)
|
||||||
|
|
||||||
|
# The migration: the toggle is on, the file is an image, the row is not.
|
||||||
|
assert importer._row_kind_mismatch(svg, text_row, _settings_(images=True)) is True
|
||||||
|
# Already an image doc → nothing to migrate.
|
||||||
|
assert importer._row_kind_mismatch(svg, image_row, _settings_(images=True)) is False
|
||||||
|
# Toggle off → the walk never admits the file, the prune guard protects
|
||||||
|
# the row: never force the image path (LOCKED A7).
|
||||||
|
assert importer._row_kind_mismatch(svg, text_row, _settings_(images=False)) is False
|
||||||
|
assert importer._row_kind_mismatch(svg, image_row, _settings_(images=False)) is False
|
||||||
|
# A file outside the image set is a text file either way.
|
||||||
|
assert importer._row_kind_mismatch(md, text_row, _settings_(images=True)) is False
|
||||||
|
# An owner-narrowed image set (no ``svg``) is not a mismatch either.
|
||||||
|
narrow = _settings(images=True, image_extensions="png,jpg")
|
||||||
|
assert importer._row_kind_mismatch(svg, text_row, narrow) is False
|
||||||
@@ -0,0 +1,426 @@
|
|||||||
|
"""Unit tests: SVG text extraction (phase 181, task 01).
|
||||||
|
|
||||||
|
:func:`app.rag.svg_text.extract_svg_text` is a PURE function — the text
|
||||||
|
that makes an SVG document retrievable WITHOUT a vision call (LOCKED
|
||||||
|
A1). No DB, no filesystem, no LLM: every case is a literal SVG string,
|
||||||
|
so the whole contract is pinned here before the importer branch (task
|
||||||
|
02) relies on it.
|
||||||
|
|
||||||
|
The matrix:
|
||||||
|
|
||||||
|
* the guards that run BEFORE any parse — empty / whitespace-only input,
|
||||||
|
the :data:`~app.rag.svg_text.MAX_SVG_BYTES` cap, the
|
||||||
|
``<!DOCTYPE``/``<!ENTITY`` refusal (case-insensitive, both a doctype
|
||||||
|
wrapper and a bare entity declaration), malformed XML → ``None``
|
||||||
|
(never a raise);
|
||||||
|
* extraction — ``<title>``/``<desc>``/``<text>`` in DOCUMENT order,
|
||||||
|
``<tspan>`` children joined with a space, namespace-prefixed and
|
||||||
|
namespaced tags treated alike, ``xlink:`` attributes ignored,
|
||||||
|
whitespace normalised, repeats dropped case-insensitively keeping the
|
||||||
|
first, ``&``/``—`` decoded by the parser;
|
||||||
|
* the :data:`~app.rag.svg_text.MAX_SVG_TEXT_CHARS` cap — cut at the
|
||||||
|
last newline/period before the cap, hard-cut only when the window
|
||||||
|
holds no boundary at all;
|
||||||
|
* the LOCKED A2 structural fallback — stem, shape census (counts,
|
||||||
|
plural, top-3 selection with a document-order tie-break) and the
|
||||||
|
``viewBox`` clause, each omitted when the file does not state it.
|
||||||
|
|
||||||
|
Importer-level tests (the vector branch, the no-vision-call seam) live
|
||||||
|
in the phase's ingest module (tasks 02/05).
|
||||||
|
"""
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from app.rag.svg_text import (
|
||||||
|
MAX_SVG_BYTES,
|
||||||
|
MAX_SVG_TEXT_CHARS,
|
||||||
|
SvgText,
|
||||||
|
extract_svg_text,
|
||||||
|
local_name,
|
||||||
|
)
|
||||||
|
|
||||||
|
SVG_NS = "http://www.w3.org/2000/svg"
|
||||||
|
XLINK_NS = "http://www.w3.org/1999/xlink"
|
||||||
|
|
||||||
|
|
||||||
|
# --- constants -------------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
def test_module_constants_are_pinned() -> None:
|
||||||
|
"""The caps are part of the contract the importer's tests assume."""
|
||||||
|
assert MAX_SVG_BYTES == 2 * 1024 * 1024
|
||||||
|
assert MAX_SVG_TEXT_CHARS == 4000
|
||||||
|
|
||||||
|
|
||||||
|
# --- the local_name helper -------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
def test_local_name_strips_the_clark_notation_namespace() -> None:
|
||||||
|
assert local_name(f"{{{SVG_NS}}}title") == "title"
|
||||||
|
assert local_name("title") == "title"
|
||||||
|
|
||||||
|
|
||||||
|
def test_local_name_of_a_non_string_tag_is_empty() -> None:
|
||||||
|
"""Comment / PI markers are callables — they carry no name."""
|
||||||
|
assert local_name(object()) == ""
|
||||||
|
|
||||||
|
|
||||||
|
# --- guards (no parse attempted) -------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
def test_empty_input_is_a_refusal_not_a_crash() -> None:
|
||||||
|
assert extract_svg_text(b"") is None
|
||||||
|
assert extract_svg_text(b"", stem="diagram") is None
|
||||||
|
|
||||||
|
|
||||||
|
def test_whitespace_only_input_is_a_refusal() -> None:
|
||||||
|
assert extract_svg_text(b" \n\t \r\n") is None
|
||||||
|
|
||||||
|
|
||||||
|
def test_bytes_over_the_size_cap_are_refused() -> None:
|
||||||
|
"""The memory guard: an over-cap file never reaches the parser."""
|
||||||
|
padded = b"<svg><title>" + b"x" * (MAX_SVG_BYTES + 1) + b"</title></svg>"
|
||||||
|
assert len(padded) > MAX_SVG_BYTES
|
||||||
|
assert extract_svg_text(padded, stem="huge") is None
|
||||||
|
|
||||||
|
|
||||||
|
def test_bytes_exactly_at_the_size_cap_are_parsed() -> None:
|
||||||
|
"""The cap is ``>``-based: exactly ``MAX_SVG_BYTES`` is in range."""
|
||||||
|
opener, closer = b"<svg><title>", b"</title></svg>"
|
||||||
|
filler = b"x" * (MAX_SVG_BYTES - len(opener) - len(closer))
|
||||||
|
result = extract_svg_text(opener + filler + closer)
|
||||||
|
assert result is not None
|
||||||
|
assert result.structural is False
|
||||||
|
|
||||||
|
|
||||||
|
def test_doctype_with_an_internal_entity_is_refused() -> None:
|
||||||
|
"""Stdlib ElementTree happily EXPANDS an internal entity — proving
|
||||||
|
why the doctype-bearing input is refused outright (billion-laughs
|
||||||
|
class): the reference must never be resolved."""
|
||||||
|
svg = b'<!DOCTYPE svg [<!ENTITY x "y">]><svg><title>&x;</title></svg>'
|
||||||
|
assert extract_svg_text(svg, stem="bomb") is None
|
||||||
|
|
||||||
|
|
||||||
|
def test_bare_entity_declaration_is_refused() -> None:
|
||||||
|
assert extract_svg_text(b'<!ENTITY x "y"><svg><title>x</title></svg>') is None
|
||||||
|
|
||||||
|
|
||||||
|
def test_entity_declaration_hidden_inside_content_is_refused() -> None:
|
||||||
|
"""The marker scan is over the raw bytes — a declaration anywhere
|
||||||
|
(even in a position the parser would reject) refuses the file."""
|
||||||
|
assert extract_svg_text(b"<svg>\n<!entity x 'y'>\n</svg>") is None
|
||||||
|
|
||||||
|
|
||||||
|
def test_guards_are_case_insensitive() -> None:
|
||||||
|
assert extract_svg_text(b'<!doctype svg SYSTEM "http://x/y.dtd"><svg/>') is None
|
||||||
|
assert extract_svg_text(b'<!entity x "y"><svg/>') is None
|
||||||
|
|
||||||
|
|
||||||
|
def test_malformed_xml_is_a_refusal_not_an_exception() -> None:
|
||||||
|
"""The caller's fail-soft is a skip (``images_failed``), so a parse
|
||||||
|
error must never escape."""
|
||||||
|
assert extract_svg_text(b"<svg><title>unclosed") is None
|
||||||
|
assert extract_svg_text(b"<svg><rect></svg>") is None
|
||||||
|
assert extract_svg_text(b"\x00\x01\x02") is None
|
||||||
|
assert extract_svg_text(b"not xml at all") is None
|
||||||
|
|
||||||
|
|
||||||
|
# --- extraction: title / desc / text ---------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
def test_title_desc_and_text_are_collected_in_document_order() -> None:
|
||||||
|
svg = (
|
||||||
|
b"<svg>"
|
||||||
|
b"<title>Top Title</title>"
|
||||||
|
b"<desc>What the diagram shows</desc>"
|
||||||
|
b"<text>first label</text>"
|
||||||
|
b"<text>second label</text>"
|
||||||
|
b"</svg>"
|
||||||
|
)
|
||||||
|
assert extract_svg_text(svg, stem="topo") == SvgText(
|
||||||
|
text="Top Title\nWhat the diagram shows\nfirst label\nsecond label",
|
||||||
|
structural=False,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def test_a_title_after_its_shape_still_follows_document_order() -> None:
|
||||||
|
svg = b"<svg><text>later</text><g><title>group title</title></g><desc>d</desc></svg>"
|
||||||
|
assert extract_svg_text(svg) == SvgText(
|
||||||
|
text="later\ngroup title\nd", structural=False
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def test_title_and_desc_concatenate_nested_markup() -> None:
|
||||||
|
svg = b"<title>Deploy <tspan>flow</tspan> map</title>"
|
||||||
|
full = b"<svg>" + svg + b"</svg>"
|
||||||
|
assert extract_svg_text(full) == SvgText(text="Deploy flow map", structural=False)
|
||||||
|
|
||||||
|
|
||||||
|
def test_text_children_are_joined_with_a_space() -> None:
|
||||||
|
"""``<text>CPU<tspan>usage</tspan></text>`` reads "CPU usage" —
|
||||||
|
without the space the labels weld into one unsearchable token."""
|
||||||
|
svg = b'<svg><text>CPU<tspan>usage</tspan>7d</text></svg>'
|
||||||
|
assert extract_svg_text(svg) == SvgText(text="CPU usage 7d", structural=False)
|
||||||
|
|
||||||
|
|
||||||
|
def test_whitespace_is_normalised_in_every_item() -> None:
|
||||||
|
svg = b"<svg><title> rack\n\t layout map </title><text> a\n b </text></svg>"
|
||||||
|
assert extract_svg_text(svg) == SvgText(
|
||||||
|
text="rack layout map\na b", structural=False
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def test_blank_title_is_dropped_and_never_stops_the_walk() -> None:
|
||||||
|
svg = b"<svg><title> </title><desc> </desc><text>real label</text></svg>"
|
||||||
|
assert extract_svg_text(svg) == SvgText(text="real label", structural=False)
|
||||||
|
|
||||||
|
|
||||||
|
def test_only_blank_text_falls_back_to_the_structural_line() -> None:
|
||||||
|
"""A blank ``<title>``/``<text>`` contributes no words, so the file
|
||||||
|
is still described by what it is (the empty ``<text>`` counts as a
|
||||||
|
drawn element)."""
|
||||||
|
svg = b'<svg viewBox="0 0 4 4"><title></title><text> </text></svg>'
|
||||||
|
assert extract_svg_text(svg, stem="blank") == SvgText(
|
||||||
|
text='Vector diagram "blank" — 1 shape (text), viewBox 4×4', structural=True
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def test_repeats_are_dropped_case_insensitively_keeping_the_first() -> None:
|
||||||
|
svg = (
|
||||||
|
b"<svg>"
|
||||||
|
b"<desc>Power Draw</desc>"
|
||||||
|
b"<title>power draw</title>"
|
||||||
|
b"<text>Other</text>"
|
||||||
|
b"</svg>"
|
||||||
|
)
|
||||||
|
assert extract_svg_text(svg) == SvgText(text="Power Draw\nOther", structural=False)
|
||||||
|
|
||||||
|
|
||||||
|
def test_parser_decodes_entity_references_in_the_text() -> None:
|
||||||
|
"""``&`` comes out decoded — the renderer re-escapes on display
|
||||||
|
(phase 180), so storing the decoded value is the correct direction."""
|
||||||
|
svg = b"<svg><title>Red & Blue — map</title></svg>"
|
||||||
|
assert extract_svg_text(svg) == SvgText(text="Red & Blue — map", structural=False)
|
||||||
|
|
||||||
|
|
||||||
|
def test_attributes_are_never_text() -> None:
|
||||||
|
svg = (
|
||||||
|
b'<svg xmlns:xlink="'
|
||||||
|
+ XLINK_NS.encode()
|
||||||
|
+ b'"><title lang="en">Visible</title>'
|
||||||
|
b'<image xlink:href="secret-internal-name.png"/></svg>'
|
||||||
|
)
|
||||||
|
assert extract_svg_text(svg, stem="diagram") == SvgText(
|
||||||
|
text="Visible", structural=False
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def test_an_image_only_svg_yields_no_words_from_its_href() -> None:
|
||||||
|
"""ASSUMPTION (task 01): an embedded ``<image>`` is not followed —
|
||||||
|
its ``href`` is not text, so the structural line describes the file."""
|
||||||
|
svg = (
|
||||||
|
b'<svg xmlns:xlink="'
|
||||||
|
+ XLINK_NS.encode()
|
||||||
|
+ b'" viewBox="0 0 10 10">'
|
||||||
|
b'<image xlink:href="inner.png"/>'
|
||||||
|
b"</svg>"
|
||||||
|
)
|
||||||
|
result = extract_svg_text(svg, stem="poster")
|
||||||
|
assert result is not None
|
||||||
|
assert result.structural is True
|
||||||
|
assert result.text == 'Vector diagram "poster" — 1 shape (image), viewBox 10×10'
|
||||||
|
|
||||||
|
|
||||||
|
# --- extraction: namespaces ------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
def test_the_default_svg_namespace_is_ignored() -> None:
|
||||||
|
svg = (
|
||||||
|
b'<svg xmlns="'
|
||||||
|
+ SVG_NS.encode()
|
||||||
|
+ b'"><title>Namespace title</title><desc>ns desc</desc>'
|
||||||
|
b'<text>ns label</text></svg>'
|
||||||
|
)
|
||||||
|
assert extract_svg_text(svg) == SvgText(
|
||||||
|
text="Namespace title\nns desc\nns label", structural=False
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def test_a_declared_prefix_form_counts_like_the_plain_tag() -> None:
|
||||||
|
"""``ElementTree`` expands ``<svg:title>`` to Clark notation, so the
|
||||||
|
local-name match covers it for free."""
|
||||||
|
svg = (
|
||||||
|
b'<svg xmlns:svg="'
|
||||||
|
+ SVG_NS.encode()
|
||||||
|
+ b'"><svg:title>Prefixed</svg:title>'
|
||||||
|
b"<svg:desc>Prefixed desc</svg:desc></svg>"
|
||||||
|
)
|
||||||
|
assert extract_svg_text(svg) == SvgText(
|
||||||
|
text="Prefixed\nPrefixed desc", structural=False
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def test_a_foreign_namespace_element_is_not_worded_text() -> None:
|
||||||
|
"""A ``<title>`` in a NON-SVG vocabulary is still matched by local
|
||||||
|
name (the walk is namespace-agnostic) while the foreign element
|
||||||
|
counts as a shape when it carries no words."""
|
||||||
|
svg = (
|
||||||
|
b'<svg xmlns:h="http://www.w3.org/1999/xhtml">'
|
||||||
|
b'<foreignObject><h:p>paragraph</h:p></foreignObject>'
|
||||||
|
b"</svg>"
|
||||||
|
)
|
||||||
|
result = extract_svg_text(svg, stem="mixed")
|
||||||
|
assert result is not None
|
||||||
|
assert result.structural is True
|
||||||
|
assert result.text == 'Vector diagram "mixed" — 2 shapes (foreignObject, p)'
|
||||||
|
|
||||||
|
|
||||||
|
def test_the_xml_declaration_encoding_is_honoured() -> None:
|
||||||
|
svg = b"<?xml version='1.0' encoding='UTF-8'?><svg><title>Caf\xc3\xa9</title></svg>"
|
||||||
|
assert extract_svg_text(svg) == SvgText(text="Café", structural=False)
|
||||||
|
|
||||||
|
|
||||||
|
def test_comments_are_not_text_and_not_shapes() -> None:
|
||||||
|
svg = b'<svg viewBox="0 0 2 2"><!-- a comment --><path d="M0 0"/></svg>'
|
||||||
|
result = extract_svg_text(svg, stem="clean")
|
||||||
|
assert result is not None
|
||||||
|
assert result.text == 'Vector diagram "clean" — 1 shape (path), viewBox 2×2'
|
||||||
|
|
||||||
|
|
||||||
|
# --- the MAX_SVG_TEXT_CHARS cap --------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
def test_extraction_under_the_cap_is_untouched() -> None:
|
||||||
|
svg = b"<svg><title>" + b"x" * (MAX_SVG_TEXT_CHARS - 20) + b"</title></svg>"
|
||||||
|
result = extract_svg_text(svg)
|
||||||
|
assert result is not None
|
||||||
|
assert len(result.text) == MAX_SVG_TEXT_CHARS - 20
|
||||||
|
|
||||||
|
|
||||||
|
def test_over_cap_text_ends_on_a_sentence_boundary() -> None:
|
||||||
|
svg = b"<svg><title>" + b"x. " * 2000 + b"</title></svg>"
|
||||||
|
result = extract_svg_text(svg)
|
||||||
|
assert result is not None
|
||||||
|
assert result.structural is False
|
||||||
|
assert len(result.text) <= MAX_SVG_TEXT_CHARS
|
||||||
|
assert len(result.text) > MAX_SVG_TEXT_CHARS - 10
|
||||||
|
assert result.text.endswith(".")
|
||||||
|
assert result.text.count("x.") < 2000 # the cut is mid-item
|
||||||
|
|
||||||
|
|
||||||
|
def test_over_cap_text_prefers_the_last_newline_boundary() -> None:
|
||||||
|
"""Two items joined with ``\\n``: the newline is the LAST boundary
|
||||||
|
inside the window, so the second item is dropped whole (never a
|
||||||
|
partial word) and the trailing newline is stripped."""
|
||||||
|
first = b"a" * (MAX_SVG_TEXT_CHARS - 10)
|
||||||
|
svg = b"<svg><title>" + first + b"</title><text>" + b"b" * 200 + b"</text></svg>"
|
||||||
|
result = extract_svg_text(svg)
|
||||||
|
assert result is not None
|
||||||
|
assert result.text == first.decode()
|
||||||
|
|
||||||
|
|
||||||
|
def test_over_cap_text_without_any_boundary_is_hard_cut() -> None:
|
||||||
|
svg = b"<svg><title>" + b"x" * (MAX_SVG_TEXT_CHARS + 1000) + b"</title></svg>"
|
||||||
|
result = extract_svg_text(svg)
|
||||||
|
assert result is not None
|
||||||
|
assert result.text == "x" * MAX_SVG_TEXT_CHARS
|
||||||
|
|
||||||
|
|
||||||
|
# --- the structural fallback (LOCKED A2) -----------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
def test_single_shape_line_matches_the_pinned_wording() -> None:
|
||||||
|
svg = b'<svg><path d="M0 0 L1 1"/></svg>'
|
||||||
|
assert extract_svg_text(svg, stem="topo") == SvgText(
|
||||||
|
text='Vector diagram "topo" — 1 shape (path)', structural=True
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def test_shape_noun_pluralises_and_counts_are_listed() -> None:
|
||||||
|
svg = b"<svg><path/><path/><path/></svg>"
|
||||||
|
assert extract_svg_text(svg, stem="topo") == SvgText(
|
||||||
|
text='Vector diagram "topo" — 3 shapes (path×3)', structural=True
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def test_top_three_kinds_with_document_order_tie_break() -> None:
|
||||||
|
svg = b"<svg><g/><g/><g/><path/><path/><circle/><rect/></svg>"
|
||||||
|
result = extract_svg_text(svg, stem="census")
|
||||||
|
assert result is not None
|
||||||
|
assert result.text == 'Vector diagram "census" — 7 shapes (g×3, path×2, circle)'
|
||||||
|
assert result.structural is True
|
||||||
|
|
||||||
|
|
||||||
|
def test_structure_elements_are_never_counted_as_shapes() -> None:
|
||||||
|
svg = b"<svg><title> </title><desc/><g><rect/></g></svg>"
|
||||||
|
result = extract_svg_text(svg, stem="nested")
|
||||||
|
assert result is not None
|
||||||
|
assert result.text == 'Vector diagram "nested" — 2 shapes (g, rect)'
|
||||||
|
|
||||||
|
|
||||||
|
def test_a_nested_svg_element_is_structure_too() -> None:
|
||||||
|
svg = b"<svg><g><svg><rect/></svg></g></svg>"
|
||||||
|
result = extract_svg_text(svg, stem="inset")
|
||||||
|
assert result is not None
|
||||||
|
assert result.text == 'Vector diagram "inset" — 2 shapes (g, rect)'
|
||||||
|
|
||||||
|
|
||||||
|
def test_viewbox_reports_width_and_height() -> None:
|
||||||
|
svg = b'<svg viewBox="0 0 128 64"><path/></svg>'
|
||||||
|
result = extract_svg_text(svg, stem="wide")
|
||||||
|
assert result is not None
|
||||||
|
assert result.text == 'Vector diagram "wide" — 1 shape (path), viewBox 128×64'
|
||||||
|
|
||||||
|
|
||||||
|
def test_viewbox_accepts_commas_and_integral_floats() -> None:
|
||||||
|
svg = b'<svg viewBox=" 0 , 0 , 8.0 , 6.0 "><path/></svg>'
|
||||||
|
result = extract_svg_text(svg, stem="comma")
|
||||||
|
assert result is not None
|
||||||
|
assert result.text == 'Vector diagram "comma" — 1 shape (path), viewBox 8×6'
|
||||||
|
|
||||||
|
|
||||||
|
def test_viewbox_keeps_fractional_sizes_without_padding() -> None:
|
||||||
|
svg = b'<svg viewBox="0 0 8.5 6.25"><g><circle/></g></svg>'
|
||||||
|
result = extract_svg_text(svg)
|
||||||
|
assert result is not None
|
||||||
|
assert result.text == "Vector diagram — 2 shapes (g, circle), viewBox 8.5×6.25"
|
||||||
|
|
||||||
|
|
||||||
|
def test_unparseable_viewbox_omits_the_clause() -> None:
|
||||||
|
for raw in (b"", b"0 0 8", b"0 0 8 6 4", b"0 0 wide tall", b"0 0 NaN NaN"):
|
||||||
|
svg = b'<svg viewBox="' + raw + b'"><path/></svg>'
|
||||||
|
result = extract_svg_text(svg, stem="odd")
|
||||||
|
assert result is not None, raw
|
||||||
|
assert result.text == 'Vector diagram "odd" — 1 shape (path)', raw
|
||||||
|
|
||||||
|
|
||||||
|
def test_missing_viewbox_omits_the_clause() -> None:
|
||||||
|
result = extract_svg_text(b"<svg><path/></svg>", stem="nobox")
|
||||||
|
assert result is not None
|
||||||
|
assert "viewBox" not in result.text
|
||||||
|
|
||||||
|
|
||||||
|
def test_viewbox_without_shapes_is_the_only_fact() -> None:
|
||||||
|
svg = b'<svg viewBox="0 0 16 9"></svg>'
|
||||||
|
assert extract_svg_text(svg, stem="canvas") == SvgText(
|
||||||
|
text='Vector diagram "canvas" — viewBox 16×9', structural=True
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def test_empty_stem_never_renders_empty_quotes() -> None:
|
||||||
|
assert extract_svg_text(b"<svg></svg>") == SvgText(
|
||||||
|
text="Vector diagram", structural=True
|
||||||
|
)
|
||||||
|
assert extract_svg_text(b'<svg viewBox="0 0 1 1"><path/></svg>') == SvgText(
|
||||||
|
text="Vector diagram — 1 shape (path), viewBox 1×1", structural=True
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def test_a_root_that_is_not_svg_still_yields_honest_text() -> None:
|
||||||
|
"""The walk is tag-based — a mis-named root is not rejected, and the
|
||||||
|
root is never counted among the shapes."""
|
||||||
|
result = extract_svg_text(b"<html><title>Legacy</title></html>", stem="legacy")
|
||||||
|
assert result == SvgText(text="Legacy", structural=False)
|
||||||
|
structural = extract_svg_text(b"<html><body/></html>", stem="legacy")
|
||||||
|
assert structural == SvgText(
|
||||||
|
text='Vector diagram "legacy" — 1 shape (body)', structural=True
|
||||||
|
)
|
||||||
Reference in New Issue
Block a user