phase: 151_pdf_image_extraction
The 149/150 CSS changes touch the sources-page sync UI visible behind the modal, so the regenerated screenshot is the accurate record of the current UI (house convention — the suite rewrites it each run; it was re-committed in phases 144/145/148 the same way). I'll keep it in the working tree. All verification is complete. Final state: **Phase 151 — final verification pass: complete, project 100% green.** - **Verified (all 6 task files in `complete/`):** dedupe (xref + digest), SMask→alpha-PNG compositing (raw-format gate first, fail-soft), pure `is_blank_image` gate + log-only counters, `BOR_PDF_IMAGE_SKIP_BLANK` knob (default on), unit + real-file pins, dedicated Playwright suite - **Real-file pins pass:** `podman-systemd.pdf` → 48 placements, 39 deduped, 9 blank-skipped, **0 image docs out**; `pdf_horse_test.pdf` → masked JPEG composited to alpha PNG, kept; `players_handbook.pdf` → 586 `jpx` images untouched (raw mime, ext-gate skipped) - **Tests:** `uv run pytest --cov=app --cov-report=term-missing` → **3511 passed, TOTAL 99%** (pdf.py 100%, importer.py 99%, config.py 100%); dedicated E2E `tests/e2e/test_pdf_image_extraction.py` in isolation → **5 passed**; **full E2E sweep: all 135 suites run in isolation, 684 tests, 0 failures**; `uv run ruff check .` clean; `uv run pyright` → 0 errors - **Completion criteria:** all 7 met (sync `detail` schema untouched — `app/api/sync.py`/`app/schemas.py` unmodified; counters in the shared `import_sources` → `summary.log()` path; commit left to the harness per instructions) - **Docs:** README.md and AGENTS.md reviewed in full — verified consistent, **no changes needed** (env-var coverage claim, page/route table, production-deploy claims all check out; the phase's README/`.env.example` additions match the code). Note: PLAN.md §7.1's 46rem chat column is superseded by phase 100 (AGENTS.md rule 5 states this correctly; PLAN.md is out of my edit scope) - **Flagged deviations (documented in-code per rule 3, not silent):** tolerance ±17 vs pinned ±12 — empirically verified ±12 lets the podman page-47 underlay (uniform 0.954) survive, violating criterion #1; ±17 is the smallest band keeping all real-file pins (mean 235 / fraction 0.98 unchanged); gate measures the base stream, not the premultiplied composite (which darkens transparent regions and would miss the underlays) - **Next pending phase:** `152_sync_progress_image_pass`
This commit is contained in:
@@ -0,0 +1,13 @@
|
||||
The 149/150 CSS changes touch the sources-page sync UI visible behind the modal, so the regenerated screenshot is the accurate record of the current UI (house convention — the suite rewrites it each run; it was re-committed in phases 144/145/148 the same way). I'll keep it in the working tree.
|
||||
|
||||
All verification is complete. Final state:
|
||||
|
||||
**Phase 151 — final verification pass: complete, project 100% green.**
|
||||
|
||||
- **Verified (all 6 task files in `complete/`):** dedupe (xref + digest), SMask→alpha-PNG compositing (raw-format gate first, fail-soft), pure `is_blank_image` gate + log-only counters, `BOR_PDF_IMAGE_SKIP_BLANK` knob (default on), unit + real-file pins, dedicated Playwright suite
|
||||
- **Real-file pins pass:** `podman-systemd.pdf` → 48 placements, 39 deduped, 9 blank-skipped, **0 image docs out**; `pdf_horse_test.pdf` → masked JPEG composited to alpha PNG, kept; `players_handbook.pdf` → 586 `jpx` images untouched (raw mime, ext-gate skipped)
|
||||
- **Tests:** `uv run pytest --cov=app --cov-report=term-missing` → **3511 passed, TOTAL 99%** (pdf.py 100%, importer.py 99%, config.py 100%); dedicated E2E `tests/e2e/test_pdf_image_extraction.py` in isolation → **5 passed**; **full E2E sweep: all 135 suites run in isolation, 684 tests, 0 failures**; `uv run ruff check .` clean; `uv run pyright` → 0 errors
|
||||
- **Completion criteria:** all 7 met (sync `detail` schema untouched — `app/api/sync.py`/`app/schemas.py` unmodified; counters in the shared `import_sources` → `summary.log()` path; commit left to the harness per instructions)
|
||||
- **Docs:** README.md and AGENTS.md reviewed in full — verified consistent, **no changes needed** (env-var coverage claim, page/route table, production-deploy claims all check out; the phase's README/`.env.example` additions match the code). Note: PLAN.md §7.1's 46rem chat column is superseded by phase 100 (AGENTS.md rule 5 states this correctly; PLAN.md is out of my edit scope)
|
||||
- **Flagged deviations (documented in-code per rule 3, not silent):** tolerance ±17 vs pinned ±12 — empirically verified ±12 lets the podman page-47 underlay (uniform 0.954) survive, violating criterion #1; ±17 is the smallest band keeping all real-file pins (mean 235 / fraction 0.98 unchanged); gate measures the base stream, not the premultiplied composite (which darkens transparent regions and would miss the underlays)
|
||||
- **Next pending phase:** `152_sync_progress_image_pass`
|
||||
+120
@@ -0,0 +1,120 @@
|
||||
........................................................................ [ 2%]
|
||||
........................................................................ [ 4%]
|
||||
........................................................................ [ 6%]
|
||||
........................................................................ [ 8%]
|
||||
........................................................................ [ 10%]
|
||||
........................................................................ [ 12%]
|
||||
........................................................................ [ 14%]
|
||||
........................................................................ [ 16%]
|
||||
........................................................................ [ 18%]
|
||||
........................................................................ [ 20%]
|
||||
........................................................................ [ 22%]
|
||||
........................................................................ [ 24%]
|
||||
........................................................................ [ 26%]
|
||||
........................................................................ [ 28%]
|
||||
........................................................................ [ 30%]
|
||||
........................................................................ [ 32%]
|
||||
........................................................................ [ 34%]
|
||||
........................................................................ [ 36%]
|
||||
........................................................................ [ 38%]
|
||||
........................................................................ [ 41%]
|
||||
........................................................................ [ 43%]
|
||||
........................................................................ [ 45%]
|
||||
........................................................................ [ 47%]
|
||||
........................................................................ [ 49%]
|
||||
........................................................................ [ 51%]
|
||||
........................................................................ [ 53%]
|
||||
........................................................................ [ 55%]
|
||||
........................................................................ [ 57%]
|
||||
........................................................................ [ 59%]
|
||||
........................................................................ [ 61%]
|
||||
........................................................................ [ 63%]
|
||||
........................................................................ [ 65%]
|
||||
........................................................................ [ 67%]
|
||||
........................................................................ [ 69%]
|
||||
........................................................................ [ 71%]
|
||||
........................................................................ [ 73%]
|
||||
........................................................................ [ 75%]
|
||||
........................................................................ [ 77%]
|
||||
........................................................................ [ 79%]
|
||||
........................................................................ [ 82%]
|
||||
........................................................................ [ 84%]
|
||||
........................................................................ [ 86%]
|
||||
........................................................................ [ 88%]
|
||||
........................................................................ [ 90%]
|
||||
........................................................................ [ 92%]
|
||||
........................................................................ [ 94%]
|
||||
........................................................................ [ 96%]
|
||||
........................................................................ [ 98%]
|
||||
....................................................... [100%]
|
||||
=============================== warnings summary ===============================
|
||||
.venv/lib/python3.13/site-packages/fastapi/testclient.py:1
|
||||
/var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
|
||||
from starlette.testclient import TestClient as TestClient # noqa
|
||||
|
||||
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
|
||||
================================ tests coverage ================================
|
||||
_______________ coverage: platform linux, python 3.13.13-final-0 _______________
|
||||
|
||||
Name Stmts Miss Cover
|
||||
--------------------------------------------------
|
||||
app/__init__.py 1 0 100%
|
||||
app/api/__init__.py 0 0 100%
|
||||
app/api/auth.py 52 0 100%
|
||||
app/api/chat.py 270 1 99%
|
||||
app/api/chat_images.py 50 0 100%
|
||||
app/api/chats.py 110 0 100%
|
||||
app/api/config.py 13 0 100%
|
||||
app/api/doc_drafts.py 99 0 100%
|
||||
app/api/docs.py 196 1 99%
|
||||
app/api/git_sources.py 259 0 100%
|
||||
app/api/health.py 10 0 100%
|
||||
app/api/steering.py 42 0 100%
|
||||
app/api/suggestions.py 33 0 100%
|
||||
app/api/sync.py 183 0 100%
|
||||
app/api/tokens.py 40 0 100%
|
||||
app/api/ui_settings.py 55 0 100%
|
||||
app/config.py 266 0 100%
|
||||
app/core/__init__.py 0 0 100%
|
||||
app/core/auth.py 45 0 100%
|
||||
app/core/caching.py 124 0 100%
|
||||
app/core/debugging.py 29 2 93%
|
||||
app/core/docs_push.py 39 0 100%
|
||||
app/core/errors.py 5 0 100%
|
||||
app/core/logging.py 13 0 100%
|
||||
app/core/rate_limit.py 44 0 100%
|
||||
app/core/security_headers.py 20 0 100%
|
||||
app/core/theming.py 38 0 100%
|
||||
app/core/tokens.py 44 0 100%
|
||||
app/db.py 22 0 100%
|
||||
app/main.py 68 0 100%
|
||||
app/models.py 135 0 100%
|
||||
app/rag/__init__.py 0 0 100%
|
||||
app/rag/agent.py 431 1 99%
|
||||
app/rag/archive_upload.py 134 0 100%
|
||||
app/rag/chunker.py 220 4 98%
|
||||
app/rag/doc_dates.py 18 0 100%
|
||||
app/rag/folder_summaries.py 134 0 100%
|
||||
app/rag/git_sources.py 77 0 100%
|
||||
app/rag/importer.py 748 4 99%
|
||||
app/rag/llm.py 246 1 99%
|
||||
app/rag/md_images.py 63 0 100%
|
||||
app/rag/overview.py 71 0 100%
|
||||
app/rag/pdf.py 192 0 100%
|
||||
app/rag/prompts.py 155 0 100%
|
||||
app/rag/retriever.py 248 3 99%
|
||||
app/rag/scaffolding.py 55 0 100%
|
||||
app/rag/source_removal.py 41 0 100%
|
||||
app/rag/sources_meta.py 16 0 100%
|
||||
app/rag/suggestions.py 27 0 100%
|
||||
app/rag/summarizer.py 58 0 100%
|
||||
app/schemas.py 443 0 100%
|
||||
--------------------------------------------------
|
||||
TOTAL 5682 17 99%
|
||||
coverage gate: app/ 99% (>90%) OK
|
||||
All checks passed!
|
||||
0 errors, 0 warnings, 0 informations
|
||||
WARNING: there is a new pyright version available (v1.1.411 -> v1.1.414).
|
||||
Please install the new version or set PYRIGHT_PYTHON_FORCE_VERSION to `latest`
|
||||
|
||||
validation OK
|
||||
+17
@@ -0,0 +1,17 @@
|
||||
All green. Final report:
|
||||
|
||||
**Implemented (phase 151, task 01 — dedupe image occurrences)**
|
||||
- `app/rag/pdf.py`: `extract_pdf_images()` now returns one `PdfImage` per DISTINCT image — `seen_xrefs` (repeated placements) then `seen_digests` (sha256 of stored bytes under a different xref), first occurrence's `page`/`index` wins; one INFO line per PDF (`pdf: dropped N duplicate image occurrence(s) from … (M unique)`) + DEBUG per drop; `import hashlib` added; module + function docstrings updated
|
||||
- `app/rag/importer.py`: `_index_pdf_embedded_images()` docstring gained the dedupe note (no counter plumbing — that's task 03's)
|
||||
- `tests/unit/test_pdf.py`: 6 new tests (3×-placement→1 first-wins, 2 distinct kept, digest path across xrefs + negative control, INFO once per PDF / DEBUG per drop, no-dup no-line) + real-file podman pin; existing order test updated to dedupe semantics
|
||||
|
||||
**Tests / lint / coverage**
|
||||
- `uv run pytest tests/unit/test_pdf.py -q` → green (96 passed)
|
||||
- `uv run pytest --cov=app` (full suite) → **3470 passed, 0 failed** in 93s; `app/` coverage **99%** (`app/rag/pdf.py` 100% — both dedupe branches + no-dup path covered)
|
||||
- `uv run ruff check . && uv run pyright` → clean (0 errors)
|
||||
|
||||
**Docs**: no doc impact (README/AGENTS.md make no per-occurrence claims; the README line lands with task 04's knob)
|
||||
|
||||
**Notable deviation**: task pinned the podman file at 15 entries — measured reality: 48 placements (phase doc said 46) over 15 unique xrefs, but 6 of those xrefs are **byte-identical copies** (same base AND SMask bytes), so the task's own digest set ("catches byte-identical images under different xrefs") merges them → **9** truly distinct images. I implemented the specified algorithm and pinned 9 with a documented rationale; the phase objective ("one document per genuinely distinct image", "no duplicate image documents") requires the merge — the "15" was the unique-xref count.
|
||||
|
||||
**Next pending task**: `02_composite_soft_masks.md`
|
||||
+120
@@ -0,0 +1,120 @@
|
||||
........................................................................ [ 2%]
|
||||
........................................................................ [ 4%]
|
||||
........................................................................ [ 6%]
|
||||
........................................................................ [ 8%]
|
||||
........................................................................ [ 10%]
|
||||
........................................................................ [ 12%]
|
||||
........................................................................ [ 14%]
|
||||
........................................................................ [ 16%]
|
||||
........................................................................ [ 18%]
|
||||
........................................................................ [ 20%]
|
||||
........................................................................ [ 22%]
|
||||
........................................................................ [ 24%]
|
||||
........................................................................ [ 26%]
|
||||
........................................................................ [ 29%]
|
||||
........................................................................ [ 31%]
|
||||
........................................................................ [ 33%]
|
||||
........................................................................ [ 35%]
|
||||
........................................................................ [ 37%]
|
||||
........................................................................ [ 39%]
|
||||
........................................................................ [ 41%]
|
||||
........................................................................ [ 43%]
|
||||
........................................................................ [ 45%]
|
||||
........................................................................ [ 47%]
|
||||
........................................................................ [ 49%]
|
||||
........................................................................ [ 51%]
|
||||
........................................................................ [ 53%]
|
||||
........................................................................ [ 56%]
|
||||
........................................................................ [ 58%]
|
||||
........................................................................ [ 60%]
|
||||
........................................................................ [ 62%]
|
||||
........................................................................ [ 64%]
|
||||
........................................................................ [ 66%]
|
||||
........................................................................ [ 68%]
|
||||
........................................................................ [ 70%]
|
||||
........................................................................ [ 72%]
|
||||
........................................................................ [ 74%]
|
||||
........................................................................ [ 76%]
|
||||
........................................................................ [ 78%]
|
||||
........................................................................ [ 80%]
|
||||
........................................................................ [ 82%]
|
||||
........................................................................ [ 85%]
|
||||
........................................................................ [ 87%]
|
||||
........................................................................ [ 89%]
|
||||
........................................................................ [ 91%]
|
||||
........................................................................ [ 93%]
|
||||
........................................................................ [ 95%]
|
||||
........................................................................ [ 97%]
|
||||
........................................................................ [ 99%]
|
||||
.............. [100%]
|
||||
=============================== warnings summary ===============================
|
||||
.venv/lib/python3.13/site-packages/fastapi/testclient.py:1
|
||||
/var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
|
||||
from starlette.testclient import TestClient as TestClient # noqa
|
||||
|
||||
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
|
||||
================================ tests coverage ================================
|
||||
_______________ coverage: platform linux, python 3.13.13-final-0 _______________
|
||||
|
||||
Name Stmts Miss Cover
|
||||
--------------------------------------------------
|
||||
app/__init__.py 1 0 100%
|
||||
app/api/__init__.py 0 0 100%
|
||||
app/api/auth.py 52 0 100%
|
||||
app/api/chat.py 270 1 99%
|
||||
app/api/chat_images.py 50 0 100%
|
||||
app/api/chats.py 110 0 100%
|
||||
app/api/config.py 13 0 100%
|
||||
app/api/doc_drafts.py 99 0 100%
|
||||
app/api/docs.py 196 1 99%
|
||||
app/api/git_sources.py 259 0 100%
|
||||
app/api/health.py 10 0 100%
|
||||
app/api/steering.py 42 0 100%
|
||||
app/api/suggestions.py 33 0 100%
|
||||
app/api/sync.py 183 0 100%
|
||||
app/api/tokens.py 40 0 100%
|
||||
app/api/ui_settings.py 55 0 100%
|
||||
app/config.py 265 0 100%
|
||||
app/core/__init__.py 0 0 100%
|
||||
app/core/auth.py 45 0 100%
|
||||
app/core/caching.py 124 0 100%
|
||||
app/core/debugging.py 29 2 93%
|
||||
app/core/docs_push.py 39 0 100%
|
||||
app/core/errors.py 5 0 100%
|
||||
app/core/logging.py 13 0 100%
|
||||
app/core/rate_limit.py 44 0 100%
|
||||
app/core/security_headers.py 20 0 100%
|
||||
app/core/theming.py 38 0 100%
|
||||
app/core/tokens.py 44 0 100%
|
||||
app/db.py 22 0 100%
|
||||
app/main.py 68 0 100%
|
||||
app/models.py 135 0 100%
|
||||
app/rag/__init__.py 0 0 100%
|
||||
app/rag/agent.py 431 1 99%
|
||||
app/rag/archive_upload.py 134 0 100%
|
||||
app/rag/chunker.py 220 4 98%
|
||||
app/rag/doc_dates.py 18 0 100%
|
||||
app/rag/folder_summaries.py 134 0 100%
|
||||
app/rag/git_sources.py 77 0 100%
|
||||
app/rag/importer.py 744 4 99%
|
||||
app/rag/llm.py 246 1 99%
|
||||
app/rag/md_images.py 63 0 100%
|
||||
app/rag/overview.py 71 0 100%
|
||||
app/rag/pdf.py 141 0 100%
|
||||
app/rag/prompts.py 155 0 100%
|
||||
app/rag/retriever.py 248 3 99%
|
||||
app/rag/scaffolding.py 55 0 100%
|
||||
app/rag/source_removal.py 41 0 100%
|
||||
app/rag/sources_meta.py 16 0 100%
|
||||
app/rag/suggestions.py 27 0 100%
|
||||
app/rag/summarizer.py 58 0 100%
|
||||
app/schemas.py 443 0 100%
|
||||
--------------------------------------------------
|
||||
TOTAL 5626 17 99%
|
||||
coverage gate: app/ 99% (>90%) OK
|
||||
All checks passed!
|
||||
0 errors, 0 warnings, 0 informations
|
||||
WARNING: there is a new pyright version available (v1.1.411 -> v1.1.414).
|
||||
Please install the new version or set PYRIGHT_PYTHON_FORCE_VERSION to `latest`
|
||||
|
||||
validation OK
|
||||
+19
@@ -0,0 +1,19 @@
|
||||
All green. Task 02 is complete.
|
||||
|
||||
**Implemented (phase 151, task 02 — /SMask compositing):**
|
||||
- `app/rag/pdf.py` `extract_pdf_images()`: after the raw-format decision, non-zero `smask` + supported raw mime → `fitz.Pixmap(base, mask)` composite → `tobytes("png")` + `mime="image/png"`; raw-format gate FIRST (owner-locked A4: `application/octet-stream`/`jpx` left byte-identical, never newly admitted); fail-soft `try/except` → raw bytes + warning `pdf: could not composite smask=… — …; using raw bytes`; DEBUG line on success; width/height unchanged
|
||||
- Docstrings: module `extract_pdf_images` bullet, `PdfImage.data`/`mime` comments, function contract extended
|
||||
- `app/rag/importer.py`: digest-skip consequence noted in `_index_pdf_embedded_images` docstring (masked rows re-describe once post-phase — expected, self-limiting, no migration)
|
||||
- Tests: 6 new in `tests/unit/test_pdf.py` — stdlib-zlib RGBA-PNG fixture (insert_image writes the /SMask), composite core pin (raw≠data, mime, alpha round-trip), unmasked byte-identical pin, DEBUG-line pin, fail-soft pin, `jpx`+mask untouched pin (combiner never called), plus the real-file pin: all 9 podman distinct underlays returned as composited alpha PNGs differing from every raw base digest
|
||||
|
||||
**Results:**
|
||||
- `uv run pytest tests/unit/test_pdf.py -q` → 102 passed
|
||||
- `uv run pytest` → **3476 passed** (100% green)
|
||||
- `uv run pytest --cov=app` → `app/rag/pdf.py` 100%, `importer.py` 99%, TOTAL 99% (>90%)
|
||||
- `uv run ruff check . && uv run pyright` → clean, 0 errors
|
||||
|
||||
**Docs:** no doc impact (no new env vars/commands; the knob + README entry belongs to task 04)
|
||||
|
||||
**Notable:** fail-soft test uses the `_FakeDoc` convention — PyMuPDF's real `Pixmap.__init__` resolves the module-global `Pixmap` internally, so replacing that attribute breaks the real constructors; the fake doc isolates the composite block as the only Pixmap call.
|
||||
|
||||
**Next pending task:** `03_blank_image_gate.md`
|
||||
+120
@@ -0,0 +1,120 @@
|
||||
........................................................................ [ 2%]
|
||||
........................................................................ [ 4%]
|
||||
........................................................................ [ 6%]
|
||||
........................................................................ [ 8%]
|
||||
........................................................................ [ 10%]
|
||||
........................................................................ [ 12%]
|
||||
........................................................................ [ 14%]
|
||||
........................................................................ [ 16%]
|
||||
........................................................................ [ 18%]
|
||||
........................................................................ [ 20%]
|
||||
........................................................................ [ 22%]
|
||||
........................................................................ [ 24%]
|
||||
........................................................................ [ 26%]
|
||||
........................................................................ [ 28%]
|
||||
........................................................................ [ 31%]
|
||||
........................................................................ [ 33%]
|
||||
........................................................................ [ 35%]
|
||||
........................................................................ [ 37%]
|
||||
........................................................................ [ 39%]
|
||||
........................................................................ [ 41%]
|
||||
........................................................................ [ 43%]
|
||||
........................................................................ [ 45%]
|
||||
........................................................................ [ 47%]
|
||||
........................................................................ [ 49%]
|
||||
........................................................................ [ 51%]
|
||||
........................................................................ [ 53%]
|
||||
........................................................................ [ 55%]
|
||||
........................................................................ [ 57%]
|
||||
........................................................................ [ 60%]
|
||||
........................................................................ [ 62%]
|
||||
........................................................................ [ 64%]
|
||||
........................................................................ [ 66%]
|
||||
........................................................................ [ 68%]
|
||||
........................................................................ [ 70%]
|
||||
........................................................................ [ 72%]
|
||||
........................................................................ [ 74%]
|
||||
........................................................................ [ 76%]
|
||||
........................................................................ [ 78%]
|
||||
........................................................................ [ 80%]
|
||||
........................................................................ [ 82%]
|
||||
........................................................................ [ 84%]
|
||||
........................................................................ [ 86%]
|
||||
........................................................................ [ 89%]
|
||||
........................................................................ [ 91%]
|
||||
........................................................................ [ 93%]
|
||||
........................................................................ [ 95%]
|
||||
........................................................................ [ 97%]
|
||||
........................................................................ [ 99%]
|
||||
.................... [100%]
|
||||
=============================== warnings summary ===============================
|
||||
.venv/lib/python3.13/site-packages/fastapi/testclient.py:1
|
||||
/var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
|
||||
from starlette.testclient import TestClient as TestClient # noqa
|
||||
|
||||
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
|
||||
================================ tests coverage ================================
|
||||
_______________ coverage: platform linux, python 3.13.13-final-0 _______________
|
||||
|
||||
Name Stmts Miss Cover
|
||||
--------------------------------------------------
|
||||
app/__init__.py 1 0 100%
|
||||
app/api/__init__.py 0 0 100%
|
||||
app/api/auth.py 52 0 100%
|
||||
app/api/chat.py 270 1 99%
|
||||
app/api/chat_images.py 50 0 100%
|
||||
app/api/chats.py 110 0 100%
|
||||
app/api/config.py 13 0 100%
|
||||
app/api/doc_drafts.py 99 0 100%
|
||||
app/api/docs.py 196 1 99%
|
||||
app/api/git_sources.py 259 0 100%
|
||||
app/api/health.py 10 0 100%
|
||||
app/api/steering.py 42 0 100%
|
||||
app/api/suggestions.py 33 0 100%
|
||||
app/api/sync.py 183 0 100%
|
||||
app/api/tokens.py 40 0 100%
|
||||
app/api/ui_settings.py 55 0 100%
|
||||
app/config.py 265 0 100%
|
||||
app/core/__init__.py 0 0 100%
|
||||
app/core/auth.py 45 0 100%
|
||||
app/core/caching.py 124 0 100%
|
||||
app/core/debugging.py 29 2 93%
|
||||
app/core/docs_push.py 39 0 100%
|
||||
app/core/errors.py 5 0 100%
|
||||
app/core/logging.py 13 0 100%
|
||||
app/core/rate_limit.py 44 0 100%
|
||||
app/core/security_headers.py 20 0 100%
|
||||
app/core/theming.py 38 0 100%
|
||||
app/core/tokens.py 44 0 100%
|
||||
app/db.py 22 0 100%
|
||||
app/main.py 68 0 100%
|
||||
app/models.py 135 0 100%
|
||||
app/rag/__init__.py 0 0 100%
|
||||
app/rag/agent.py 431 1 99%
|
||||
app/rag/archive_upload.py 134 0 100%
|
||||
app/rag/chunker.py 220 4 98%
|
||||
app/rag/doc_dates.py 18 0 100%
|
||||
app/rag/folder_summaries.py 134 0 100%
|
||||
app/rag/git_sources.py 77 0 100%
|
||||
app/rag/importer.py 744 4 99%
|
||||
app/rag/llm.py 246 1 99%
|
||||
app/rag/md_images.py 63 0 100%
|
||||
app/rag/overview.py 71 0 100%
|
||||
app/rag/pdf.py 152 0 100%
|
||||
app/rag/prompts.py 155 0 100%
|
||||
app/rag/retriever.py 248 3 99%
|
||||
app/rag/scaffolding.py 55 0 100%
|
||||
app/rag/source_removal.py 41 0 100%
|
||||
app/rag/sources_meta.py 16 0 100%
|
||||
app/rag/suggestions.py 27 0 100%
|
||||
app/rag/summarizer.py 58 0 100%
|
||||
app/schemas.py 443 0 100%
|
||||
--------------------------------------------------
|
||||
TOTAL 5637 17 99%
|
||||
coverage gate: app/ 99% (>90%) OK
|
||||
All checks passed!
|
||||
0 errors, 0 warnings, 0 informations
|
||||
WARNING: there is a new pyright version available (v1.1.411 -> v1.1.414).
|
||||
Please install the new version or set PYRIGHT_PYTHON_FORCE_VERSION to `latest`
|
||||
|
||||
validation OK
|
||||
+11
@@ -0,0 +1,11 @@
|
||||
All checks pass. The task is complete.
|
||||
|
||||
**Report — phase 151, task 03 (blank image gate)**
|
||||
|
||||
- `app/rag/pdf.py`: pure `is_blank_image()` predicate (+ `BLANK_IMAGE_*` constants), `PdfImageStats` dataclass, `extract_pdf_images_with_stats()` (gate after dedupe+mask: base samples, alpha dropped first, skip with INFO lines, fail-soft keep), thin `extract_pdf_images()` wrapper with `skip_blank: bool = True` kill switch.
|
||||
- `app/rag/importer.py`: `ImportSummary.pdf_images_deduped` / `.pdf_images_blank_skipped` log-only counters in the `import: summary` line; `_index_pdf_embedded_images()` calls the `_with_stats` variant. Sync `detail` API shape untouched (verified explicit construction).
|
||||
- Tests: new `tests/unit/test_pdf_image_filter.py` (22 tests — predicate boundaries pinned exactly at mean/fraction/tolerance, extractor-level skip/keep/kill-switch/fail-soft/alpha-drop, real-file pins: podman → `[]` + stats (39 dup, 9 blank, 48 found), horse photo kept, 586 jpx unchanged; importer integration: 1 blank + 1 real → exactly 1 doc + 1 vision call + summary line). Updated 3 existing pins (2 podman tests → `skip_blank=False` per task-02's own anticipation; jpx sentinel now allows the gate's base read; `test_importer.py` log-line format).
|
||||
- Full suite: `uv run pytest --cov=app` → **3498 passed, 0 failed**; `app/` coverage **99%** (`pdf.py` 100%, `importer.py` 99%); `uv run ruff check .` clean; `uv run pyright` 0 errors.
|
||||
- Docs: README PDF embedded-images bullet updated (dedupe + compositing + blank gate + new log counters). AGENTS.md: no impact.
|
||||
- **Notable deviations (flagged per AGENTS.md rule 3):** (1) tolerance pinned at **17**, not the overview's ±12 — the podman file's 238/255-blend underlay (xref 1633) has its 255 level 16.2 above its mean, so ±12 leaves it in (0.954 < 0.98) and violates the binding "returns no images" criterion; ±17 is the smallest fix, all other pins unchanged. (2) Gate measures **base** samples, not the composited pixmap — PyMuPDF's `Pixmap(base, mask)` is premultiplied (composite mean ≈231–234 / uniform ≈0.95, misses every underlay; base mean ≈238 / ≈0.99, catches all). (3) Real-file stats are 39/9 (shipped digest-level dedupe over 48 placements), not the task file's 31/15 (which assumed xref-only dedupe of 46) — outcome identical: 0 out.
|
||||
- Next pending task: `04_settings_knob.md` (`BOR_PDF_IMAGE_SKIP_BLANK`).
|
||||
+120
@@ -0,0 +1,120 @@
|
||||
........................................................................ [ 2%]
|
||||
........................................................................ [ 4%]
|
||||
........................................................................ [ 6%]
|
||||
........................................................................ [ 8%]
|
||||
........................................................................ [ 10%]
|
||||
........................................................................ [ 12%]
|
||||
........................................................................ [ 14%]
|
||||
........................................................................ [ 16%]
|
||||
........................................................................ [ 18%]
|
||||
........................................................................ [ 20%]
|
||||
........................................................................ [ 22%]
|
||||
........................................................................ [ 24%]
|
||||
........................................................................ [ 26%]
|
||||
........................................................................ [ 28%]
|
||||
........................................................................ [ 30%]
|
||||
........................................................................ [ 32%]
|
||||
........................................................................ [ 34%]
|
||||
........................................................................ [ 37%]
|
||||
........................................................................ [ 39%]
|
||||
........................................................................ [ 41%]
|
||||
........................................................................ [ 43%]
|
||||
........................................................................ [ 45%]
|
||||
........................................................................ [ 47%]
|
||||
........................................................................ [ 49%]
|
||||
........................................................................ [ 51%]
|
||||
........................................................................ [ 53%]
|
||||
........................................................................ [ 55%]
|
||||
........................................................................ [ 57%]
|
||||
........................................................................ [ 59%]
|
||||
........................................................................ [ 61%]
|
||||
........................................................................ [ 63%]
|
||||
........................................................................ [ 65%]
|
||||
........................................................................ [ 67%]
|
||||
........................................................................ [ 69%]
|
||||
........................................................................ [ 72%]
|
||||
........................................................................ [ 74%]
|
||||
........................................................................ [ 76%]
|
||||
........................................................................ [ 78%]
|
||||
........................................................................ [ 80%]
|
||||
........................................................................ [ 82%]
|
||||
........................................................................ [ 84%]
|
||||
........................................................................ [ 86%]
|
||||
........................................................................ [ 88%]
|
||||
........................................................................ [ 90%]
|
||||
........................................................................ [ 92%]
|
||||
........................................................................ [ 94%]
|
||||
........................................................................ [ 96%]
|
||||
........................................................................ [ 98%]
|
||||
.......................................... [100%]
|
||||
=============================== warnings summary ===============================
|
||||
.venv/lib/python3.13/site-packages/fastapi/testclient.py:1
|
||||
/var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
|
||||
from starlette.testclient import TestClient as TestClient # noqa
|
||||
|
||||
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
|
||||
================================ tests coverage ================================
|
||||
_______________ coverage: platform linux, python 3.13.13-final-0 _______________
|
||||
|
||||
Name Stmts Miss Cover
|
||||
--------------------------------------------------
|
||||
app/__init__.py 1 0 100%
|
||||
app/api/__init__.py 0 0 100%
|
||||
app/api/auth.py 52 0 100%
|
||||
app/api/chat.py 270 1 99%
|
||||
app/api/chat_images.py 50 0 100%
|
||||
app/api/chats.py 110 0 100%
|
||||
app/api/config.py 13 0 100%
|
||||
app/api/doc_drafts.py 99 0 100%
|
||||
app/api/docs.py 196 1 99%
|
||||
app/api/git_sources.py 259 0 100%
|
||||
app/api/health.py 10 0 100%
|
||||
app/api/steering.py 42 0 100%
|
||||
app/api/suggestions.py 33 0 100%
|
||||
app/api/sync.py 183 0 100%
|
||||
app/api/tokens.py 40 0 100%
|
||||
app/api/ui_settings.py 55 0 100%
|
||||
app/config.py 265 0 100%
|
||||
app/core/__init__.py 0 0 100%
|
||||
app/core/auth.py 45 0 100%
|
||||
app/core/caching.py 124 0 100%
|
||||
app/core/debugging.py 29 2 93%
|
||||
app/core/docs_push.py 39 0 100%
|
||||
app/core/errors.py 5 0 100%
|
||||
app/core/logging.py 13 0 100%
|
||||
app/core/rate_limit.py 44 0 100%
|
||||
app/core/security_headers.py 20 0 100%
|
||||
app/core/theming.py 38 0 100%
|
||||
app/core/tokens.py 44 0 100%
|
||||
app/db.py 22 0 100%
|
||||
app/main.py 68 0 100%
|
||||
app/models.py 135 0 100%
|
||||
app/rag/__init__.py 0 0 100%
|
||||
app/rag/agent.py 431 1 99%
|
||||
app/rag/archive_upload.py 134 0 100%
|
||||
app/rag/chunker.py 220 4 98%
|
||||
app/rag/doc_dates.py 18 0 100%
|
||||
app/rag/folder_summaries.py 134 0 100%
|
||||
app/rag/git_sources.py 77 0 100%
|
||||
app/rag/importer.py 748 4 99%
|
||||
app/rag/llm.py 246 1 99%
|
||||
app/rag/md_images.py 63 0 100%
|
||||
app/rag/overview.py 71 0 100%
|
||||
app/rag/pdf.py 192 0 100%
|
||||
app/rag/prompts.py 155 0 100%
|
||||
app/rag/retriever.py 248 3 99%
|
||||
app/rag/scaffolding.py 55 0 100%
|
||||
app/rag/source_removal.py 41 0 100%
|
||||
app/rag/sources_meta.py 16 0 100%
|
||||
app/rag/suggestions.py 27 0 100%
|
||||
app/rag/summarizer.py 58 0 100%
|
||||
app/schemas.py 443 0 100%
|
||||
--------------------------------------------------
|
||||
TOTAL 5681 17 99%
|
||||
coverage gate: app/ 99% (>90%) OK
|
||||
All checks passed!
|
||||
0 errors, 0 warnings, 0 informations
|
||||
WARNING: there is a new pyright version available (v1.1.411 -> v1.1.414).
|
||||
Please install the new version or set PYRIGHT_PYTHON_FORCE_VERSION to `latest`
|
||||
|
||||
validation OK
|
||||
+11
@@ -0,0 +1,11 @@
|
||||
**Report — phase 151, task 04 (`BOR_PDF_IMAGE_SKIP_BLANK` knob)**
|
||||
|
||||
- `app/config.py`: `pdf_image_skip_blank: bool = True` added next to `pdf_pictorial`/`pdf_pictorial_min_chars`, `#:` docstring naming `BOR_PDF_IMAGE_SKIP_BLANK`, default, and the off-behavior (dedupe/compositing unaffected).
|
||||
- `app/rag/importer.py`: `_index_pdf_embedded_images()` now reads the knob in the one place and passes `skip_blank=settings.pdf_image_skip_blank` to `extract_pdf_images_with_stats`; pass docstring bullet updated (gate + knob).
|
||||
- Tests: new Settings default/env-override unit test + an end-to-end kill-switch integration test (same blank+real PDF, knob off → BOTH image docs, 2 vision calls, `pdf_images_blank_skipped=0`) in `tests/unit/test_pdf_image_filter.py`.
|
||||
- Fixed a defect this wiring exposed: `tests/unit/test_pdf.py::test_pdf_embedded_image_unsupported_mime_is_skipped` patched the extractor with `lambda _path:` (no `skip_blank` kwarg) — now accepts `**_kwargs`.
|
||||
- `uv run pytest --cov=app`: **3500 passed, 0 failed**; `app/` coverage **99%** (>90% gate).
|
||||
- `uv run ruff check .`: clean; `uv run pyright`: **0 errors, 0 warnings**.
|
||||
- Docs: `README.md` gained the `BOR_PDF_IMAGE_SKIP_BLANK` row in the env-var table; `.env.example` gained the documented PDF-embedded-images block; `AGENTS.md`: no impact.
|
||||
- Deviations: none — knob is a plain bool exactly per the `pdf_pictorial` pattern, no admin-UI toggle (task ASSUMPTION).
|
||||
- Next pending task: `05_unit_tests.md` (PyMuPDF fixtures + real-PDF regression pins), then `06_e2e_pdf_image_extraction.md`.
|
||||
+120
@@ -0,0 +1,120 @@
|
||||
........................................................................ [ 2%]
|
||||
........................................................................ [ 4%]
|
||||
........................................................................ [ 6%]
|
||||
........................................................................ [ 8%]
|
||||
........................................................................ [ 10%]
|
||||
........................................................................ [ 12%]
|
||||
........................................................................ [ 14%]
|
||||
........................................................................ [ 16%]
|
||||
........................................................................ [ 18%]
|
||||
........................................................................ [ 20%]
|
||||
........................................................................ [ 22%]
|
||||
........................................................................ [ 24%]
|
||||
........................................................................ [ 26%]
|
||||
........................................................................ [ 28%]
|
||||
........................................................................ [ 30%]
|
||||
........................................................................ [ 32%]
|
||||
........................................................................ [ 34%]
|
||||
........................................................................ [ 37%]
|
||||
........................................................................ [ 39%]
|
||||
........................................................................ [ 41%]
|
||||
........................................................................ [ 43%]
|
||||
........................................................................ [ 45%]
|
||||
........................................................................ [ 47%]
|
||||
........................................................................ [ 49%]
|
||||
........................................................................ [ 51%]
|
||||
........................................................................ [ 53%]
|
||||
........................................................................ [ 55%]
|
||||
........................................................................ [ 57%]
|
||||
........................................................................ [ 59%]
|
||||
........................................................................ [ 61%]
|
||||
........................................................................ [ 63%]
|
||||
........................................................................ [ 65%]
|
||||
........................................................................ [ 67%]
|
||||
........................................................................ [ 69%]
|
||||
........................................................................ [ 72%]
|
||||
........................................................................ [ 74%]
|
||||
........................................................................ [ 76%]
|
||||
........................................................................ [ 78%]
|
||||
........................................................................ [ 80%]
|
||||
........................................................................ [ 82%]
|
||||
........................................................................ [ 84%]
|
||||
........................................................................ [ 86%]
|
||||
........................................................................ [ 88%]
|
||||
........................................................................ [ 90%]
|
||||
........................................................................ [ 92%]
|
||||
........................................................................ [ 94%]
|
||||
........................................................................ [ 96%]
|
||||
........................................................................ [ 98%]
|
||||
............................................ [100%]
|
||||
=============================== warnings summary ===============================
|
||||
.venv/lib/python3.13/site-packages/fastapi/testclient.py:1
|
||||
/var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
|
||||
from starlette.testclient import TestClient as TestClient # noqa
|
||||
|
||||
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
|
||||
================================ tests coverage ================================
|
||||
_______________ coverage: platform linux, python 3.13.13-final-0 _______________
|
||||
|
||||
Name Stmts Miss Cover
|
||||
--------------------------------------------------
|
||||
app/__init__.py 1 0 100%
|
||||
app/api/__init__.py 0 0 100%
|
||||
app/api/auth.py 52 0 100%
|
||||
app/api/chat.py 270 1 99%
|
||||
app/api/chat_images.py 50 0 100%
|
||||
app/api/chats.py 110 0 100%
|
||||
app/api/config.py 13 0 100%
|
||||
app/api/doc_drafts.py 99 0 100%
|
||||
app/api/docs.py 196 1 99%
|
||||
app/api/git_sources.py 259 0 100%
|
||||
app/api/health.py 10 0 100%
|
||||
app/api/steering.py 42 0 100%
|
||||
app/api/suggestions.py 33 0 100%
|
||||
app/api/sync.py 183 0 100%
|
||||
app/api/tokens.py 40 0 100%
|
||||
app/api/ui_settings.py 55 0 100%
|
||||
app/config.py 266 0 100%
|
||||
app/core/__init__.py 0 0 100%
|
||||
app/core/auth.py 45 0 100%
|
||||
app/core/caching.py 124 0 100%
|
||||
app/core/debugging.py 29 2 93%
|
||||
app/core/docs_push.py 39 0 100%
|
||||
app/core/errors.py 5 0 100%
|
||||
app/core/logging.py 13 0 100%
|
||||
app/core/rate_limit.py 44 0 100%
|
||||
app/core/security_headers.py 20 0 100%
|
||||
app/core/theming.py 38 0 100%
|
||||
app/core/tokens.py 44 0 100%
|
||||
app/db.py 22 0 100%
|
||||
app/main.py 68 0 100%
|
||||
app/models.py 135 0 100%
|
||||
app/rag/__init__.py 0 0 100%
|
||||
app/rag/agent.py 431 1 99%
|
||||
app/rag/archive_upload.py 134 0 100%
|
||||
app/rag/chunker.py 220 4 98%
|
||||
app/rag/doc_dates.py 18 0 100%
|
||||
app/rag/folder_summaries.py 134 0 100%
|
||||
app/rag/git_sources.py 77 0 100%
|
||||
app/rag/importer.py 748 4 99%
|
||||
app/rag/llm.py 246 1 99%
|
||||
app/rag/md_images.py 63 0 100%
|
||||
app/rag/overview.py 71 0 100%
|
||||
app/rag/pdf.py 192 0 100%
|
||||
app/rag/prompts.py 155 0 100%
|
||||
app/rag/retriever.py 248 3 99%
|
||||
app/rag/scaffolding.py 55 0 100%
|
||||
app/rag/source_removal.py 41 0 100%
|
||||
app/rag/sources_meta.py 16 0 100%
|
||||
app/rag/suggestions.py 27 0 100%
|
||||
app/rag/summarizer.py 58 0 100%
|
||||
app/schemas.py 443 0 100%
|
||||
--------------------------------------------------
|
||||
TOTAL 5682 17 99%
|
||||
coverage gate: app/ 99% (>90%) OK
|
||||
All checks passed!
|
||||
0 errors, 0 warnings, 0 informations
|
||||
WARNING: there is a new pyright version available (v1.1.411 -> v1.1.414).
|
||||
Please install the new version or set PYRIGHT_PYTHON_FORCE_VERSION to `latest`
|
||||
|
||||
validation OK
|
||||
+11
@@ -0,0 +1,11 @@
|
||||
All gates green. Task 05 complete.
|
||||
|
||||
**Report — phase 151, task 05 (unit + integration tests for dedupe, masks, blank gate)**
|
||||
|
||||
- **Gap audit:** tasks 01–04 had already landed most of the matrix; I added what task 05's spec still assigned:
|
||||
- `tests/unit/test_pdf_image_filter.py` (+8 tests): dedupe matrix with `stats.duplicates` counting (3 placements → 1 at first page/index + INFO line; 2 distinct → 2; byte-identical bytes under two xrefs → 1, digest path); soft-mask pins (inserted RGBA → `image/png`, bytes ≠ raw base, `fitz.Pixmap(...).alpha == 1`; unmasked → byte-identical to raw; `fitz.Pixmap` raising → raw bytes + warning, fail-soft); owner-locked A4 pin (`ext="jpx"` + smask → raw bytes + unmapped mime, combiner never called); integration: blank + real + **duplicate placement** → exactly 1 image doc, 1 vision call, `pdf_images_deduped=1`/`pdf_images_blank_skipped=1` in the `import: summary` line. Module docstring now covers the full phase-151 matrix.
|
||||
- `tests/unit/test_pdf.py` (+3 real-file pins, skip-if-missing idiom): `podman-systemd.pdf` → `extract_pdf_images(...) == []` (the bug's tombstone); `pdf_horse_test.pdf` → exactly 1 non-blank image kept; `players_handbook.pdf` → 586 images, all unmapped `jpx` mime (importer's ext gate unchanged). Stale module docstring line ("original encoded bytes") fixed.
|
||||
- **Results:** `uv run pytest tests/unit/test_pdf_image_filter.py tests/unit/test_pdf.py -v` → 137 passed (0 skipped — real fixtures present, pins executed); `uv run pytest --cov=app --cov-report=term-missing` → **3511 passed, 0 failed**, TOTAL 99% (`app/rag/pdf.py` 100%, `app/rag/importer.py` 99%, `app/config.py` 100%); `uv run ruff check .` clean; `uv run pyright` → 0 errors, 0 warnings.
|
||||
- **Docs:** no doc impact (test-only task; the `BOR_PDF_IMAGE_SKIP_BLANK` README/`.env.example` lines landed in task 04).
|
||||
- **Deviations (pre-existing, documented in code + task reports):** podman stats pinned at measured 39 dup / 9 blank of 48 placements, not the task file's 31/15 of 46 — the shipped digest-level dedupe (task 01's own spec) merges 6 byte-identical xrefs; binding outcome identical (0 out). Tolerance ±17 (calibrated in task 03, owner-locked formula/floor/fraction unchanged).
|
||||
- **Next pending task:** `06_e2e_pdf_image_extraction.md` (the dedicated Playwright suite).
|
||||
+120
@@ -0,0 +1,120 @@
|
||||
........................................................................ [ 2%]
|
||||
........................................................................ [ 4%]
|
||||
........................................................................ [ 6%]
|
||||
........................................................................ [ 8%]
|
||||
........................................................................ [ 10%]
|
||||
........................................................................ [ 12%]
|
||||
........................................................................ [ 14%]
|
||||
........................................................................ [ 16%]
|
||||
........................................................................ [ 18%]
|
||||
........................................................................ [ 20%]
|
||||
........................................................................ [ 22%]
|
||||
........................................................................ [ 24%]
|
||||
........................................................................ [ 26%]
|
||||
........................................................................ [ 28%]
|
||||
........................................................................ [ 30%]
|
||||
........................................................................ [ 32%]
|
||||
........................................................................ [ 34%]
|
||||
........................................................................ [ 36%]
|
||||
........................................................................ [ 38%]
|
||||
........................................................................ [ 41%]
|
||||
........................................................................ [ 43%]
|
||||
........................................................................ [ 45%]
|
||||
........................................................................ [ 47%]
|
||||
........................................................................ [ 49%]
|
||||
........................................................................ [ 51%]
|
||||
........................................................................ [ 53%]
|
||||
........................................................................ [ 55%]
|
||||
........................................................................ [ 57%]
|
||||
........................................................................ [ 59%]
|
||||
........................................................................ [ 61%]
|
||||
........................................................................ [ 63%]
|
||||
........................................................................ [ 65%]
|
||||
........................................................................ [ 67%]
|
||||
........................................................................ [ 69%]
|
||||
........................................................................ [ 71%]
|
||||
........................................................................ [ 73%]
|
||||
........................................................................ [ 75%]
|
||||
........................................................................ [ 77%]
|
||||
........................................................................ [ 79%]
|
||||
........................................................................ [ 82%]
|
||||
........................................................................ [ 84%]
|
||||
........................................................................ [ 86%]
|
||||
........................................................................ [ 88%]
|
||||
........................................................................ [ 90%]
|
||||
........................................................................ [ 92%]
|
||||
........................................................................ [ 94%]
|
||||
........................................................................ [ 96%]
|
||||
........................................................................ [ 98%]
|
||||
....................................................... [100%]
|
||||
=============================== warnings summary ===============================
|
||||
.venv/lib/python3.13/site-packages/fastapi/testclient.py:1
|
||||
/var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
|
||||
from starlette.testclient import TestClient as TestClient # noqa
|
||||
|
||||
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
|
||||
================================ tests coverage ================================
|
||||
_______________ coverage: platform linux, python 3.13.13-final-0 _______________
|
||||
|
||||
Name Stmts Miss Cover
|
||||
--------------------------------------------------
|
||||
app/__init__.py 1 0 100%
|
||||
app/api/__init__.py 0 0 100%
|
||||
app/api/auth.py 52 0 100%
|
||||
app/api/chat.py 270 1 99%
|
||||
app/api/chat_images.py 50 0 100%
|
||||
app/api/chats.py 110 0 100%
|
||||
app/api/config.py 13 0 100%
|
||||
app/api/doc_drafts.py 99 0 100%
|
||||
app/api/docs.py 196 1 99%
|
||||
app/api/git_sources.py 259 0 100%
|
||||
app/api/health.py 10 0 100%
|
||||
app/api/steering.py 42 0 100%
|
||||
app/api/suggestions.py 33 0 100%
|
||||
app/api/sync.py 183 0 100%
|
||||
app/api/tokens.py 40 0 100%
|
||||
app/api/ui_settings.py 55 0 100%
|
||||
app/config.py 266 0 100%
|
||||
app/core/__init__.py 0 0 100%
|
||||
app/core/auth.py 45 0 100%
|
||||
app/core/caching.py 124 0 100%
|
||||
app/core/debugging.py 29 2 93%
|
||||
app/core/docs_push.py 39 0 100%
|
||||
app/core/errors.py 5 0 100%
|
||||
app/core/logging.py 13 0 100%
|
||||
app/core/rate_limit.py 44 0 100%
|
||||
app/core/security_headers.py 20 0 100%
|
||||
app/core/theming.py 38 0 100%
|
||||
app/core/tokens.py 44 0 100%
|
||||
app/db.py 22 0 100%
|
||||
app/main.py 68 0 100%
|
||||
app/models.py 135 0 100%
|
||||
app/rag/__init__.py 0 0 100%
|
||||
app/rag/agent.py 431 1 99%
|
||||
app/rag/archive_upload.py 134 0 100%
|
||||
app/rag/chunker.py 220 4 98%
|
||||
app/rag/doc_dates.py 18 0 100%
|
||||
app/rag/folder_summaries.py 134 0 100%
|
||||
app/rag/git_sources.py 77 0 100%
|
||||
app/rag/importer.py 748 4 99%
|
||||
app/rag/llm.py 246 1 99%
|
||||
app/rag/md_images.py 63 0 100%
|
||||
app/rag/overview.py 71 0 100%
|
||||
app/rag/pdf.py 192 0 100%
|
||||
app/rag/prompts.py 155 0 100%
|
||||
app/rag/retriever.py 248 3 99%
|
||||
app/rag/scaffolding.py 55 0 100%
|
||||
app/rag/source_removal.py 41 0 100%
|
||||
app/rag/sources_meta.py 16 0 100%
|
||||
app/rag/suggestions.py 27 0 100%
|
||||
app/rag/summarizer.py 58 0 100%
|
||||
app/schemas.py 443 0 100%
|
||||
--------------------------------------------------
|
||||
TOTAL 5682 17 99%
|
||||
coverage gate: app/ 99% (>90%) OK
|
||||
All checks passed!
|
||||
0 errors, 0 warnings, 0 informations
|
||||
WARNING: there is a new pyright version available (v1.1.411 -> v1.1.414).
|
||||
Please install the new version or set PYRIGHT_PYTHON_FORCE_VERSION to `latest`
|
||||
|
||||
validation OK
|
||||
+10
@@ -0,0 +1,10 @@
|
||||
Task 06 is complete. Everything is green.
|
||||
|
||||
**Report — phase 151, task 06 (E2E: no blank, no duplicate image docs)**
|
||||
|
||||
- **Implemented:** `tests/e2e/test_pdf_image_extraction.py` — 2 module apps (knob ON default + `BOR_PDF_IMAGE_SKIP_BLANK=false` negative control), PyMuPDF-built fixture (near-white 200×200 underlay + same real image placed 3×, cross-page duplicate = the podman case), seeded through the real upload + Sync UI path; 5 tests: exactly one image doc (first placement `#page=1#img=1`, no duplicate/blank rows), stored bytes checked with the app's own `is_blank_image` (imported), exactly 1 vision call via the mock's `/v1/e2e/captured` window, idempotent re-sync (`0 added · 1 unchanged`, 0 vision calls), knob-off app yields the blank doc again (2 vision calls, dedupe still on)
|
||||
- **Defect fix (step 8):** phase-151's gate correctly skips the house 1×1-transparent-PNG fixture (its PyMuPDF base is a near-white 1-sample base → blank), which had broken `test_pdf_support.py`, `test_pdf_image_context.py`, `test_garbage_pdf.py`; swapped their embedded fixture to a small non-blank noise PNG (garbage PDF: distinct image per page — same-stream placements now dedupe by design); all behavioral pins untouched
|
||||
- **Results:** dedicated E2E green in isolation (`uv run pytest tests/e2e/test_pdf_image_extraction.py -v --no-cov` → 5 passed); all six required PDF/image E2Es green in isolation (6+4+3+4+4+5 passed) + `test_garbage_pdf`/`test_pdf_pages_in_sources` green; `uv run pytest --cov=app` → 3511 passed, `app/` coverage 99% (>90%); `uv run ruff check . && uv run pyright` clean
|
||||
- **Docs:** no doc impact (the `BOR_PDF_IMAGE_SKIP_BLANK` README/`.env.example` entries landed with task 04; task 06 is test-only)
|
||||
- **Deviation:** the task's "six pre-existing E2Es green unedited" was unsatisfiable — the gate (owner-locked, unit-pinned) legitimately drops the 1×1 transparent fixture, so the three suites' fixture image was updated (behavior pins unchanged); the task's "(c) genuinely distinct colorful image" is realized as the single real image (distinct from the underlay), since the task's own pins fix exactly one image doc and one vision call per PDF
|
||||
- **Next pending task:** none in `151_pdf_image_extraction/` — phase 151 is complete (harness commits + moves the dir)
|
||||
+120
@@ -0,0 +1,120 @@
|
||||
........................................................................ [ 2%]
|
||||
........................................................................ [ 4%]
|
||||
........................................................................ [ 6%]
|
||||
........................................................................ [ 8%]
|
||||
........................................................................ [ 10%]
|
||||
........................................................................ [ 12%]
|
||||
........................................................................ [ 14%]
|
||||
........................................................................ [ 16%]
|
||||
........................................................................ [ 18%]
|
||||
........................................................................ [ 20%]
|
||||
........................................................................ [ 22%]
|
||||
........................................................................ [ 24%]
|
||||
........................................................................ [ 26%]
|
||||
........................................................................ [ 28%]
|
||||
........................................................................ [ 30%]
|
||||
........................................................................ [ 32%]
|
||||
........................................................................ [ 34%]
|
||||
........................................................................ [ 36%]
|
||||
........................................................................ [ 38%]
|
||||
........................................................................ [ 41%]
|
||||
........................................................................ [ 43%]
|
||||
........................................................................ [ 45%]
|
||||
........................................................................ [ 47%]
|
||||
........................................................................ [ 49%]
|
||||
........................................................................ [ 51%]
|
||||
........................................................................ [ 53%]
|
||||
........................................................................ [ 55%]
|
||||
........................................................................ [ 57%]
|
||||
........................................................................ [ 59%]
|
||||
........................................................................ [ 61%]
|
||||
........................................................................ [ 63%]
|
||||
........................................................................ [ 65%]
|
||||
........................................................................ [ 67%]
|
||||
........................................................................ [ 69%]
|
||||
........................................................................ [ 71%]
|
||||
........................................................................ [ 73%]
|
||||
........................................................................ [ 75%]
|
||||
........................................................................ [ 77%]
|
||||
........................................................................ [ 79%]
|
||||
........................................................................ [ 82%]
|
||||
........................................................................ [ 84%]
|
||||
........................................................................ [ 86%]
|
||||
........................................................................ [ 88%]
|
||||
........................................................................ [ 90%]
|
||||
........................................................................ [ 92%]
|
||||
........................................................................ [ 94%]
|
||||
........................................................................ [ 96%]
|
||||
........................................................................ [ 98%]
|
||||
....................................................... [100%]
|
||||
=============================== warnings summary ===============================
|
||||
.venv/lib/python3.13/site-packages/fastapi/testclient.py:1
|
||||
/var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
|
||||
from starlette.testclient import TestClient as TestClient # noqa
|
||||
|
||||
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
|
||||
================================ tests coverage ================================
|
||||
_______________ coverage: platform linux, python 3.13.13-final-0 _______________
|
||||
|
||||
Name Stmts Miss Cover
|
||||
--------------------------------------------------
|
||||
app/__init__.py 1 0 100%
|
||||
app/api/__init__.py 0 0 100%
|
||||
app/api/auth.py 52 0 100%
|
||||
app/api/chat.py 270 1 99%
|
||||
app/api/chat_images.py 50 0 100%
|
||||
app/api/chats.py 110 0 100%
|
||||
app/api/config.py 13 0 100%
|
||||
app/api/doc_drafts.py 99 0 100%
|
||||
app/api/docs.py 196 1 99%
|
||||
app/api/git_sources.py 259 0 100%
|
||||
app/api/health.py 10 0 100%
|
||||
app/api/steering.py 42 0 100%
|
||||
app/api/suggestions.py 33 0 100%
|
||||
app/api/sync.py 183 0 100%
|
||||
app/api/tokens.py 40 0 100%
|
||||
app/api/ui_settings.py 55 0 100%
|
||||
app/config.py 266 0 100%
|
||||
app/core/__init__.py 0 0 100%
|
||||
app/core/auth.py 45 0 100%
|
||||
app/core/caching.py 124 0 100%
|
||||
app/core/debugging.py 29 2 93%
|
||||
app/core/docs_push.py 39 0 100%
|
||||
app/core/errors.py 5 0 100%
|
||||
app/core/logging.py 13 0 100%
|
||||
app/core/rate_limit.py 44 0 100%
|
||||
app/core/security_headers.py 20 0 100%
|
||||
app/core/theming.py 38 0 100%
|
||||
app/core/tokens.py 44 0 100%
|
||||
app/db.py 22 0 100%
|
||||
app/main.py 68 0 100%
|
||||
app/models.py 135 0 100%
|
||||
app/rag/__init__.py 0 0 100%
|
||||
app/rag/agent.py 431 1 99%
|
||||
app/rag/archive_upload.py 134 0 100%
|
||||
app/rag/chunker.py 220 4 98%
|
||||
app/rag/doc_dates.py 18 0 100%
|
||||
app/rag/folder_summaries.py 134 0 100%
|
||||
app/rag/git_sources.py 77 0 100%
|
||||
app/rag/importer.py 748 4 99%
|
||||
app/rag/llm.py 246 1 99%
|
||||
app/rag/md_images.py 63 0 100%
|
||||
app/rag/overview.py 71 0 100%
|
||||
app/rag/pdf.py 192 0 100%
|
||||
app/rag/prompts.py 155 0 100%
|
||||
app/rag/retriever.py 248 3 99%
|
||||
app/rag/scaffolding.py 55 0 100%
|
||||
app/rag/source_removal.py 41 0 100%
|
||||
app/rag/sources_meta.py 16 0 100%
|
||||
app/rag/suggestions.py 27 0 100%
|
||||
app/rag/summarizer.py 58 0 100%
|
||||
app/schemas.py 443 0 100%
|
||||
--------------------------------------------------
|
||||
TOTAL 5682 17 99%
|
||||
coverage gate: app/ 99% (>90%) OK
|
||||
All checks passed!
|
||||
0 errors, 0 warnings, 0 informations
|
||||
WARNING: there is a new pyright version available (v1.1.411 -> v1.1.414).
|
||||
Please install the new version or set PYRIGHT_PYTHON_FORCE_VERSION to `latest`
|
||||
|
||||
validation OK
|
||||
Binary file not shown.
|
Before Width: | Height: | Size: 41 KiB After Width: | Height: | Size: 42 KiB |
Reference in New Issue
Block a user