phase: 133_pdf_screenshot_fallback
All verification is complete — everything is green. No repo-side changes were needed in this pass (the phase's work in the working tree is correct and complete).
## Phase 133 final verification report
- **Verified (already implemented in working tree):** `is_usable_pdf_text` gate (0.30 XML-ish / 0.05 non-printable / empty), `render_pdf_pages`, `_fallback_page_screenshots` + `pdfs_screenshotted` counter in `app/rag/{pdf,importer}.py`; regression pin, unit/integration/E2E suites.
- **Completion criteria:**
1. Real guide clean through real pipeline — ✅ `test_pdf_hnb_guide_regression` + E2E preview/viewer checks (0 XML-ish lines, first-page text present).
2. Garbage PDF → vision fallback end to end — ✅ unit + E2E (stored content = canned per-page descriptions, garbage not stored, sync `detail` shape unchanged).
3. Clean PDF never burns a fallback vision call — ✅ unit + E2E (byte-faithful content).
4. Gates — ✅ `uv run pytest --cov=app --cov-report=term-missing`: **3129 passed, app/ 99%** (>90%); `uv run pytest tests/e2e/test_garbage_pdf.py -v --no-cov`: **3 passed** in isolation.
5. No behavior change to completed phases — ✅ entire suite green (all prior-phase tests included); `ruff check .` + `pyright` clean.
6. Production stale row — post-deploy re-sync verification (live infra, no repo work; no live changes made — deploy of `main` is the owner's step).
- **Documentation:** README.md reviewed in full — every env default cross-checked against `app/config.py`, page routes against `app/main.py`, scripts/Containerfile/compose flags, `.env.example`; the phase-133 PDF bullet added in the working tree is accurate. AGENTS.md verified consistent (all referenced files/commands exist). **No changes needed.**
- **Next pending phase:** `134_sync_cancel`.
This commit is contained in:
+14
@@ -0,0 +1,14 @@
|
||||
All verification is complete — everything is green. No repo-side changes were needed in this pass (the phase's work in the working tree is correct and complete).
|
||||
|
||||
## Phase 133 final verification report
|
||||
|
||||
- **Verified (already implemented in working tree):** `is_usable_pdf_text` gate (0.30 XML-ish / 0.05 non-printable / empty), `render_pdf_pages`, `_fallback_page_screenshots` + `pdfs_screenshotted` counter in `app/rag/{pdf,importer}.py`; regression pin, unit/integration/E2E suites.
|
||||
- **Completion criteria:**
|
||||
1. Real guide clean through real pipeline — ✅ `test_pdf_hnb_guide_regression` + E2E preview/viewer checks (0 XML-ish lines, first-page text present).
|
||||
2. Garbage PDF → vision fallback end to end — ✅ unit + E2E (stored content = canned per-page descriptions, garbage not stored, sync `detail` shape unchanged).
|
||||
3. Clean PDF never burns a fallback vision call — ✅ unit + E2E (byte-faithful content).
|
||||
4. Gates — ✅ `uv run pytest --cov=app --cov-report=term-missing`: **3129 passed, app/ 99%** (>90%); `uv run pytest tests/e2e/test_garbage_pdf.py -v --no-cov`: **3 passed** in isolation.
|
||||
5. No behavior change to completed phases — ✅ entire suite green (all prior-phase tests included); `ruff check .` + `pyright` clean.
|
||||
6. Production stale row — post-deploy re-sync verification (live infra, no repo work; no live changes made — deploy of `main` is the owner's step).
|
||||
- **Documentation:** README.md reviewed in full — every env default cross-checked against `app/config.py`, page routes against `app/main.py`, scripts/Containerfile/compose flags, `.env.example`; the phase-133 PDF bullet added in the working tree is accurate. AGENTS.md verified consistent (all referenced files/commands exist). **No changes needed.**
|
||||
- **Next pending phase:** `134_sync_cancel`.
|
||||
+115
@@ -0,0 +1,115 @@
|
||||
........................................................................ [ 2%]
|
||||
........................................................................ [ 4%]
|
||||
........................................................................ [ 6%]
|
||||
........................................................................ [ 9%]
|
||||
........................................................................ [ 11%]
|
||||
........................................................................ [ 13%]
|
||||
........................................................................ [ 16%]
|
||||
........................................................................ [ 18%]
|
||||
........................................................................ [ 20%]
|
||||
........................................................................ [ 23%]
|
||||
........................................................................ [ 25%]
|
||||
........................................................................ [ 27%]
|
||||
........................................................................ [ 29%]
|
||||
........................................................................ [ 32%]
|
||||
........................................................................ [ 34%]
|
||||
........................................................................ [ 36%]
|
||||
........................................................................ [ 39%]
|
||||
........................................................................ [ 41%]
|
||||
........................................................................ [ 43%]
|
||||
........................................................................ [ 46%]
|
||||
........................................................................ [ 48%]
|
||||
........................................................................ [ 50%]
|
||||
........................................................................ [ 52%]
|
||||
........................................................................ [ 55%]
|
||||
........................................................................ [ 57%]
|
||||
........................................................................ [ 59%]
|
||||
........................................................................ [ 62%]
|
||||
........................................................................ [ 64%]
|
||||
........................................................................ [ 66%]
|
||||
........................................................................ [ 69%]
|
||||
........................................................................ [ 71%]
|
||||
........................................................................ [ 73%]
|
||||
........................................................................ [ 75%]
|
||||
........................................................................ [ 78%]
|
||||
........................................................................ [ 80%]
|
||||
........................................................................ [ 82%]
|
||||
........................................................................ [ 85%]
|
||||
........................................................................ [ 87%]
|
||||
........................................................................ [ 89%]
|
||||
........................................................................ [ 92%]
|
||||
........................................................................ [ 94%]
|
||||
........................................................................ [ 96%]
|
||||
........................................................................ [ 98%]
|
||||
................................. [100%]
|
||||
=============================== warnings summary ===============================
|
||||
.venv/lib/python3.13/site-packages/fastapi/testclient.py:1
|
||||
/var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
|
||||
from starlette.testclient import TestClient as TestClient # noqa
|
||||
|
||||
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
|
||||
================================ tests coverage ================================
|
||||
_______________ coverage: platform linux, python 3.13.13-final-0 _______________
|
||||
|
||||
Name Stmts Miss Cover
|
||||
--------------------------------------------------
|
||||
app/__init__.py 1 0 100%
|
||||
app/api/__init__.py 0 0 100%
|
||||
app/api/auth.py 52 0 100%
|
||||
app/api/chat.py 268 1 99%
|
||||
app/api/chat_images.py 50 0 100%
|
||||
app/api/chats.py 110 0 100%
|
||||
app/api/config.py 13 0 100%
|
||||
app/api/doc_drafts.py 99 0 100%
|
||||
app/api/docs.py 179 1 99%
|
||||
app/api/git_sources.py 241 0 100%
|
||||
app/api/health.py 10 0 100%
|
||||
app/api/steering.py 42 0 100%
|
||||
app/api/suggestions.py 33 0 100%
|
||||
app/api/sync.py 156 0 100%
|
||||
app/api/tokens.py 40 0 100%
|
||||
app/api/ui_settings.py 55 0 100%
|
||||
app/config.py 256 0 100%
|
||||
app/core/__init__.py 0 0 100%
|
||||
app/core/auth.py 45 0 100%
|
||||
app/core/caching.py 124 0 100%
|
||||
app/core/debugging.py 29 2 93%
|
||||
app/core/docs_push.py 39 0 100%
|
||||
app/core/errors.py 5 0 100%
|
||||
app/core/logging.py 13 0 100%
|
||||
app/core/rate_limit.py 44 0 100%
|
||||
app/core/security_headers.py 20 0 100%
|
||||
app/core/theming.py 38 0 100%
|
||||
app/core/tokens.py 44 0 100%
|
||||
app/db.py 22 0 100%
|
||||
app/main.py 68 0 100%
|
||||
app/models.py 132 0 100%
|
||||
app/rag/__init__.py 0 0 100%
|
||||
app/rag/agent.py 412 1 99%
|
||||
app/rag/archive_upload.py 134 0 100%
|
||||
app/rag/chunker.py 220 4 98%
|
||||
app/rag/doc_dates.py 18 0 100%
|
||||
app/rag/folder_summaries.py 134 0 100%
|
||||
app/rag/git_sources.py 38 0 100%
|
||||
app/rag/importer.py 595 9 98%
|
||||
app/rag/llm.py 246 1 99%
|
||||
app/rag/md_images.py 63 0 100%
|
||||
app/rag/overview.py 71 0 100%
|
||||
app/rag/pdf.py 109 0 100%
|
||||
app/rag/prompts.py 144 0 100%
|
||||
app/rag/retriever.py 248 3 99%
|
||||
app/rag/scaffolding.py 55 0 100%
|
||||
app/rag/source_removal.py 41 0 100%
|
||||
app/rag/sources_meta.py 16 0 100%
|
||||
app/rag/suggestions.py 27 0 100%
|
||||
app/rag/summarizer.py 49 0 100%
|
||||
app/schemas.py 410 0 100%
|
||||
--------------------------------------------------
|
||||
TOTAL 5258 22 99%
|
||||
coverage gate: app/ 99% (>90%) OK
|
||||
All checks passed!
|
||||
0 errors, 0 warnings, 0 informations
|
||||
WARNING: there is a new pyright version available (v1.1.411 -> v1.1.414).
|
||||
Please install the new version or set PYRIGHT_PYTHON_FORCE_VERSION to `latest`
|
||||
|
||||
validation OK
|
||||
+19
@@ -0,0 +1,19 @@
|
||||
Task 01 complete — all gates green on the final state.
|
||||
|
||||
**Implemented**
|
||||
- `tests/integration/test_pdf_hnb_guide_regression.py` — pins the owner's exact `test_sources/hnb_2026_guide.tar.gz` 48-page Illustrator PDF through the REAL importer: clean-content bars (XML-ish < 0.05, non-printable < 0.01), pinned first-page text + `---PAGE-1---` + stem title, 91 embedded chunks + lite summary, all 27 embedded-image docs got the fake vision reply (multimodal `chat` ≥ 1, locked bare `DESCRIBE_PROMPT`), 120 s wall-time guard; FINDING + run command + provenance in the docstring; fixture extracted to `tmp_path` only.
|
||||
|
||||
**Results**
|
||||
- `uv run pytest --cov=app --cov-report=term-missing` → 3109 passed, `app/` coverage 99% (>90%)
|
||||
- `uv run ruff check .` → all checks passed; `uv run pyright` → 0 errors
|
||||
- `uv run pytest tests/integration/test_pdf_hnb_guide_regression.py -v --no-cov` → 1 passed (0.71 s)
|
||||
- Pin has teeth: pre-126 text-branch recipe on the file measures 46.2% XML-ish / 18.6% non-printable — both bars break
|
||||
|
||||
**Docs:** no doc impact (test-only task, no user-visible change)
|
||||
|
||||
**Decisions** (deviations from the task's literal text, documented in the test docstring)
|
||||
- `summary.added == 1 + len(image_docs)` instead of `== 1`: phase-126 A3 counts each embedded-image doc `added` (code + phase-126 test confirm); `summary.files == 1` pins the walk admitted the one file. The required multimodal assertion forces `images` on, which makes image docs added.
|
||||
- `Settings(_env_file=".env", images=True, image_dir=<tmp>)`: conftest forces `BOR_IMAGES` off process-wide, so the plain form would silently skip the embedded-image pass; the kwarg restores the owner's live `BOR_IMAGES=1`, `image_dir` is hermetic per the house `_image_llm` pattern.
|
||||
- Non-printable bar excludes `\n`/`\t`/`\r` (counting them flags every document; ~2.5% of the real guide is newlines).
|
||||
|
||||
**Next pending task:** `.agents/phases/todo/133_pdf_screenshot_fallback/02_pdf_text_gate.md`
|
||||
+115
@@ -0,0 +1,115 @@
|
||||
........................................................................ [ 2%]
|
||||
........................................................................ [ 4%]
|
||||
........................................................................ [ 6%]
|
||||
........................................................................ [ 9%]
|
||||
........................................................................ [ 11%]
|
||||
........................................................................ [ 13%]
|
||||
........................................................................ [ 16%]
|
||||
........................................................................ [ 18%]
|
||||
........................................................................ [ 20%]
|
||||
........................................................................ [ 23%]
|
||||
........................................................................ [ 25%]
|
||||
........................................................................ [ 27%]
|
||||
........................................................................ [ 30%]
|
||||
........................................................................ [ 32%]
|
||||
........................................................................ [ 34%]
|
||||
........................................................................ [ 37%]
|
||||
........................................................................ [ 39%]
|
||||
........................................................................ [ 41%]
|
||||
........................................................................ [ 44%]
|
||||
........................................................................ [ 46%]
|
||||
........................................................................ [ 48%]
|
||||
........................................................................ [ 50%]
|
||||
........................................................................ [ 53%]
|
||||
........................................................................ [ 55%]
|
||||
........................................................................ [ 57%]
|
||||
........................................................................ [ 60%]
|
||||
........................................................................ [ 62%]
|
||||
........................................................................ [ 64%]
|
||||
........................................................................ [ 67%]
|
||||
........................................................................ [ 69%]
|
||||
........................................................................ [ 71%]
|
||||
........................................................................ [ 74%]
|
||||
........................................................................ [ 76%]
|
||||
........................................................................ [ 78%]
|
||||
........................................................................ [ 81%]
|
||||
........................................................................ [ 83%]
|
||||
........................................................................ [ 85%]
|
||||
........................................................................ [ 88%]
|
||||
........................................................................ [ 90%]
|
||||
........................................................................ [ 92%]
|
||||
........................................................................ [ 94%]
|
||||
........................................................................ [ 97%]
|
||||
........................................................................ [ 99%]
|
||||
............. [100%]
|
||||
=============================== warnings summary ===============================
|
||||
.venv/lib/python3.13/site-packages/fastapi/testclient.py:1
|
||||
/var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
|
||||
from starlette.testclient import TestClient as TestClient # noqa
|
||||
|
||||
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
|
||||
================================ tests coverage ================================
|
||||
_______________ coverage: platform linux, python 3.13.13-final-0 _______________
|
||||
|
||||
Name Stmts Miss Cover
|
||||
--------------------------------------------------
|
||||
app/__init__.py 1 0 100%
|
||||
app/api/__init__.py 0 0 100%
|
||||
app/api/auth.py 52 0 100%
|
||||
app/api/chat.py 268 1 99%
|
||||
app/api/chat_images.py 50 0 100%
|
||||
app/api/chats.py 110 0 100%
|
||||
app/api/config.py 13 0 100%
|
||||
app/api/doc_drafts.py 99 0 100%
|
||||
app/api/docs.py 179 1 99%
|
||||
app/api/git_sources.py 241 0 100%
|
||||
app/api/health.py 10 0 100%
|
||||
app/api/steering.py 42 0 100%
|
||||
app/api/suggestions.py 33 0 100%
|
||||
app/api/sync.py 156 0 100%
|
||||
app/api/tokens.py 40 0 100%
|
||||
app/api/ui_settings.py 55 0 100%
|
||||
app/config.py 256 0 100%
|
||||
app/core/__init__.py 0 0 100%
|
||||
app/core/auth.py 45 0 100%
|
||||
app/core/caching.py 124 0 100%
|
||||
app/core/debugging.py 29 2 93%
|
||||
app/core/docs_push.py 39 0 100%
|
||||
app/core/errors.py 5 0 100%
|
||||
app/core/logging.py 13 0 100%
|
||||
app/core/rate_limit.py 44 0 100%
|
||||
app/core/security_headers.py 20 0 100%
|
||||
app/core/theming.py 38 0 100%
|
||||
app/core/tokens.py 44 0 100%
|
||||
app/db.py 22 0 100%
|
||||
app/main.py 68 0 100%
|
||||
app/models.py 132 0 100%
|
||||
app/rag/__init__.py 0 0 100%
|
||||
app/rag/agent.py 412 1 99%
|
||||
app/rag/archive_upload.py 134 0 100%
|
||||
app/rag/chunker.py 220 4 98%
|
||||
app/rag/doc_dates.py 18 0 100%
|
||||
app/rag/folder_summaries.py 134 0 100%
|
||||
app/rag/git_sources.py 38 0 100%
|
||||
app/rag/importer.py 571 9 98%
|
||||
app/rag/llm.py 246 1 99%
|
||||
app/rag/md_images.py 63 0 100%
|
||||
app/rag/overview.py 71 0 100%
|
||||
app/rag/pdf.py 81 0 100%
|
||||
app/rag/prompts.py 144 0 100%
|
||||
app/rag/retriever.py 248 3 99%
|
||||
app/rag/scaffolding.py 55 0 100%
|
||||
app/rag/source_removal.py 41 0 100%
|
||||
app/rag/sources_meta.py 16 0 100%
|
||||
app/rag/suggestions.py 27 0 100%
|
||||
app/rag/summarizer.py 49 0 100%
|
||||
app/schemas.py 410 0 100%
|
||||
--------------------------------------------------
|
||||
TOTAL 5206 22 99%
|
||||
coverage gate: app/ 99% (>90%) OK
|
||||
All checks passed!
|
||||
0 errors, 0 warnings, 0 informations
|
||||
WARNING: there is a new pyright version available (v1.1.411 -> v1.1.414).
|
||||
Please install the new version or set PYRIGHT_PYTHON_FORCE_VERSION to `latest`
|
||||
|
||||
validation OK
|
||||
+16
@@ -0,0 +1,16 @@
|
||||
Task 02 complete — everything 100% green.
|
||||
|
||||
**Implemented**
|
||||
- `app/rag/pdf.py`: new pure, deterministic gate `is_usable_pdf_text(text) -> bool` (placed after `extract_pdf_text`) — unusable when empty text layer, ≥ `XMLISH_LINE_FRACTION` (0.30) XML-ish non-empty lines, or ≥ `NONPRINT_CHAR_FRACTION` (0.05) non-printable chars (U+FFFD + C0 controls, explicitly excluding structural `\t`/`\n`/`\r` — documented). Constants named/documented; docstring cites the FINDING (pre-126 `errors="replace"` soup, 46% XML-ish, the owner's `<illustrator:StartupProfile>` preview, ASSUMPTION A8 false-positive cost).
|
||||
- `tests/unit/test_pdf.py`: 8 new pins — clean text+separators → True; empty/whitespace → False; **the REAL pre-126 garbage** (module-scoped fixture reading the owner's tar.gz PDF with the exact old recipe, pins `is False` + both signals clearing their named bars + the `<illustrator:StartupProfile>` line present, skip-if-missing like task 01); 30% and 31% tag-like → False; 29% → True (A8 edge); U+FFFD ~7.9% soup with no tags → False (second signal alone); tab/newline-heavy text → True (exclusion).
|
||||
|
||||
**Results**
|
||||
- `uv run pytest --cov=app --cov-report=term-missing` → **3117 passed**, 0 failed (69 s); `app/rag/pdf.py` 100%, `app/` TOTAL 99% (>90%)
|
||||
- `uv run pytest tests/unit/test_pdf.py` → 64 passed
|
||||
- `uv run ruff check .` → clean; `uv run pyright` → 0 errors
|
||||
|
||||
**Docs:** no doc impact (internal predicate; not wired to the importer until task 03 — README/AGENTS.md stay accurate).
|
||||
|
||||
**Decisions:** explicit C0-range check (`ord(ch) < 32`, minus `\t\n\r`) over `unicodedata` Cc — matches the task-01 integration pin's `_nonprint_frac` convention; real-garbage fixture extracts the tar member to a module-scoped temp dir (never the repo).
|
||||
|
||||
**Next pending task:** `.agents/phases/todo/133_pdf_screenshot_fallback/03_page_screenshot_fallback.md`
|
||||
+115
@@ -0,0 +1,115 @@
|
||||
........................................................................ [ 2%]
|
||||
........................................................................ [ 4%]
|
||||
........................................................................ [ 6%]
|
||||
........................................................................ [ 9%]
|
||||
........................................................................ [ 11%]
|
||||
........................................................................ [ 13%]
|
||||
........................................................................ [ 16%]
|
||||
........................................................................ [ 18%]
|
||||
........................................................................ [ 20%]
|
||||
........................................................................ [ 23%]
|
||||
........................................................................ [ 25%]
|
||||
........................................................................ [ 27%]
|
||||
........................................................................ [ 30%]
|
||||
........................................................................ [ 32%]
|
||||
........................................................................ [ 34%]
|
||||
........................................................................ [ 36%]
|
||||
........................................................................ [ 39%]
|
||||
........................................................................ [ 41%]
|
||||
........................................................................ [ 43%]
|
||||
........................................................................ [ 46%]
|
||||
........................................................................ [ 48%]
|
||||
........................................................................ [ 50%]
|
||||
........................................................................ [ 53%]
|
||||
........................................................................ [ 55%]
|
||||
........................................................................ [ 57%]
|
||||
........................................................................ [ 60%]
|
||||
........................................................................ [ 62%]
|
||||
........................................................................ [ 64%]
|
||||
........................................................................ [ 66%]
|
||||
........................................................................ [ 69%]
|
||||
........................................................................ [ 71%]
|
||||
........................................................................ [ 73%]
|
||||
........................................................................ [ 76%]
|
||||
........................................................................ [ 78%]
|
||||
........................................................................ [ 80%]
|
||||
........................................................................ [ 83%]
|
||||
........................................................................ [ 85%]
|
||||
........................................................................ [ 87%]
|
||||
........................................................................ [ 90%]
|
||||
........................................................................ [ 92%]
|
||||
........................................................................ [ 94%]
|
||||
........................................................................ [ 97%]
|
||||
........................................................................ [ 99%]
|
||||
..................... [100%]
|
||||
=============================== warnings summary ===============================
|
||||
.venv/lib/python3.13/site-packages/fastapi/testclient.py:1
|
||||
/var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
|
||||
from starlette.testclient import TestClient as TestClient # noqa
|
||||
|
||||
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
|
||||
================================ tests coverage ================================
|
||||
_______________ coverage: platform linux, python 3.13.13-final-0 _______________
|
||||
|
||||
Name Stmts Miss Cover
|
||||
--------------------------------------------------
|
||||
app/__init__.py 1 0 100%
|
||||
app/api/__init__.py 0 0 100%
|
||||
app/api/auth.py 52 0 100%
|
||||
app/api/chat.py 268 1 99%
|
||||
app/api/chat_images.py 50 0 100%
|
||||
app/api/chats.py 110 0 100%
|
||||
app/api/config.py 13 0 100%
|
||||
app/api/doc_drafts.py 99 0 100%
|
||||
app/api/docs.py 179 1 99%
|
||||
app/api/git_sources.py 241 0 100%
|
||||
app/api/health.py 10 0 100%
|
||||
app/api/steering.py 42 0 100%
|
||||
app/api/suggestions.py 33 0 100%
|
||||
app/api/sync.py 156 0 100%
|
||||
app/api/tokens.py 40 0 100%
|
||||
app/api/ui_settings.py 55 0 100%
|
||||
app/config.py 256 0 100%
|
||||
app/core/__init__.py 0 0 100%
|
||||
app/core/auth.py 45 0 100%
|
||||
app/core/caching.py 124 0 100%
|
||||
app/core/debugging.py 29 2 93%
|
||||
app/core/docs_push.py 39 0 100%
|
||||
app/core/errors.py 5 0 100%
|
||||
app/core/logging.py 13 0 100%
|
||||
app/core/rate_limit.py 44 0 100%
|
||||
app/core/security_headers.py 20 0 100%
|
||||
app/core/theming.py 38 0 100%
|
||||
app/core/tokens.py 44 0 100%
|
||||
app/db.py 22 0 100%
|
||||
app/main.py 68 0 100%
|
||||
app/models.py 132 0 100%
|
||||
app/rag/__init__.py 0 0 100%
|
||||
app/rag/agent.py 412 1 99%
|
||||
app/rag/archive_upload.py 134 0 100%
|
||||
app/rag/chunker.py 220 4 98%
|
||||
app/rag/doc_dates.py 18 0 100%
|
||||
app/rag/folder_summaries.py 134 0 100%
|
||||
app/rag/git_sources.py 38 0 100%
|
||||
app/rag/importer.py 571 9 98%
|
||||
app/rag/llm.py 246 1 99%
|
||||
app/rag/md_images.py 63 0 100%
|
||||
app/rag/overview.py 71 0 100%
|
||||
app/rag/pdf.py 94 0 100%
|
||||
app/rag/prompts.py 144 0 100%
|
||||
app/rag/retriever.py 248 3 99%
|
||||
app/rag/scaffolding.py 55 0 100%
|
||||
app/rag/source_removal.py 41 0 100%
|
||||
app/rag/sources_meta.py 16 0 100%
|
||||
app/rag/suggestions.py 27 0 100%
|
||||
app/rag/summarizer.py 49 0 100%
|
||||
app/schemas.py 410 0 100%
|
||||
--------------------------------------------------
|
||||
TOTAL 5219 22 99%
|
||||
coverage gate: app/ 99% (>90%) OK
|
||||
All checks passed!
|
||||
0 errors, 0 warnings, 0 informations
|
||||
WARNING: there is a new pyright version available (v1.1.411 -> v1.1.414).
|
||||
Please install the new version or set PYRIGHT_PYTHON_FORCE_VERSION to `latest`
|
||||
|
||||
validation OK
|
||||
+21
@@ -0,0 +1,21 @@
|
||||
Everything is green and verified. Final report:
|
||||
|
||||
## Task 03 — page-screenshot fallback: complete
|
||||
|
||||
**Implemented**
|
||||
- `app/rag/pdf.py`: `render_pdf_pages(pdf_path, dpi=150) -> list[bytes]` — one PNG per page in page order; unreadable page → `b""` at its index (index never shifts); unopenable file → `[]`.
|
||||
- `app/rag/importer.py`: gate branch in `_index_pdf_file` after extraction (before digest/upsert) — `summary.pdfs_screenshotted += 1` + `_fallback_page_screenshots`; new helper renders pages, logs the exact loud line (`pdf: text layer unusable for <s>/<p> — falling back to N page screenshots`), vision-reads each page via phase-122 `describe_image` (fail-soft → `page N — description unavailable`), all-failed → `""` (empty-content path), `---PAGE-N---` separators; `ImportSummary.pdfs_screenshotted` + `log()` field.
|
||||
- Stale phase-126 pins updated (image-only PDF now takes the fallback; corrupt file stays empty with the fallback counted); "no extractable text" warning now fires only when the doc truly lands empty.
|
||||
|
||||
**Tests**
|
||||
- `uv run pytest --cov=app --cov-report=term-missing` → **3129 passed**, TOTAL **99%** (`app/rag/pdf.py` 100%, `importer.py` 98% — only pre-existing uncovered lines; all new code covered)
|
||||
- `uv run ruff check . && uv run pyright` → clean
|
||||
- Isolated E2E spot-checks: `test_pdf_support.py` 6 passed, `test_image_anchor_context.py` 4 passed, `test_import_documents.py` 3 passed; task-01 real-guide regression passed
|
||||
|
||||
**Docs**
|
||||
- README: PDF section gained the phase-133 fallback bullet (`pdfs_screenshotted=N`); AGENTS.md — no doc impact.
|
||||
|
||||
**Notable decision**
|
||||
- The mandated gate sits before the row lookup, so a garbage PDF's vision calls re-issue each sync before the digest check marks the row unchanged (the row itself re-imports unchanged — no re-chunk/embed/KB change). Eliminating the vision re-issue would need a schema change (out of scope); the cost is documented honestly in the docstrings instead of the overview's "no vision re-burn" phrasing. The task's optional `_gate_fractions` DEBUG line was skipped (required warning line kept exact).
|
||||
|
||||
**Next pending task:** `.agents/phases/todo/133_pdf_screenshot_fallback/04_e2e_garbage_pdf.md`
|
||||
+115
@@ -0,0 +1,115 @@
|
||||
........................................................................ [ 2%]
|
||||
........................................................................ [ 4%]
|
||||
........................................................................ [ 6%]
|
||||
........................................................................ [ 9%]
|
||||
........................................................................ [ 11%]
|
||||
........................................................................ [ 13%]
|
||||
........................................................................ [ 16%]
|
||||
........................................................................ [ 18%]
|
||||
........................................................................ [ 20%]
|
||||
........................................................................ [ 23%]
|
||||
........................................................................ [ 25%]
|
||||
........................................................................ [ 27%]
|
||||
........................................................................ [ 29%]
|
||||
........................................................................ [ 32%]
|
||||
........................................................................ [ 34%]
|
||||
........................................................................ [ 36%]
|
||||
........................................................................ [ 39%]
|
||||
........................................................................ [ 41%]
|
||||
........................................................................ [ 43%]
|
||||
........................................................................ [ 46%]
|
||||
........................................................................ [ 48%]
|
||||
........................................................................ [ 50%]
|
||||
........................................................................ [ 52%]
|
||||
........................................................................ [ 55%]
|
||||
........................................................................ [ 57%]
|
||||
........................................................................ [ 59%]
|
||||
........................................................................ [ 62%]
|
||||
........................................................................ [ 64%]
|
||||
........................................................................ [ 66%]
|
||||
........................................................................ [ 69%]
|
||||
........................................................................ [ 71%]
|
||||
........................................................................ [ 73%]
|
||||
........................................................................ [ 75%]
|
||||
........................................................................ [ 78%]
|
||||
........................................................................ [ 80%]
|
||||
........................................................................ [ 82%]
|
||||
........................................................................ [ 85%]
|
||||
........................................................................ [ 87%]
|
||||
........................................................................ [ 89%]
|
||||
........................................................................ [ 92%]
|
||||
........................................................................ [ 94%]
|
||||
........................................................................ [ 96%]
|
||||
........................................................................ [ 98%]
|
||||
................................. [100%]
|
||||
=============================== warnings summary ===============================
|
||||
.venv/lib/python3.13/site-packages/fastapi/testclient.py:1
|
||||
/var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
|
||||
from starlette.testclient import TestClient as TestClient # noqa
|
||||
|
||||
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
|
||||
================================ tests coverage ================================
|
||||
_______________ coverage: platform linux, python 3.13.13-final-0 _______________
|
||||
|
||||
Name Stmts Miss Cover
|
||||
--------------------------------------------------
|
||||
app/__init__.py 1 0 100%
|
||||
app/api/__init__.py 0 0 100%
|
||||
app/api/auth.py 52 0 100%
|
||||
app/api/chat.py 268 1 99%
|
||||
app/api/chat_images.py 50 0 100%
|
||||
app/api/chats.py 110 0 100%
|
||||
app/api/config.py 13 0 100%
|
||||
app/api/doc_drafts.py 99 0 100%
|
||||
app/api/docs.py 179 1 99%
|
||||
app/api/git_sources.py 241 0 100%
|
||||
app/api/health.py 10 0 100%
|
||||
app/api/steering.py 42 0 100%
|
||||
app/api/suggestions.py 33 0 100%
|
||||
app/api/sync.py 156 0 100%
|
||||
app/api/tokens.py 40 0 100%
|
||||
app/api/ui_settings.py 55 0 100%
|
||||
app/config.py 256 0 100%
|
||||
app/core/__init__.py 0 0 100%
|
||||
app/core/auth.py 45 0 100%
|
||||
app/core/caching.py 124 0 100%
|
||||
app/core/debugging.py 29 2 93%
|
||||
app/core/docs_push.py 39 0 100%
|
||||
app/core/errors.py 5 0 100%
|
||||
app/core/logging.py 13 0 100%
|
||||
app/core/rate_limit.py 44 0 100%
|
||||
app/core/security_headers.py 20 0 100%
|
||||
app/core/theming.py 38 0 100%
|
||||
app/core/tokens.py 44 0 100%
|
||||
app/db.py 22 0 100%
|
||||
app/main.py 68 0 100%
|
||||
app/models.py 132 0 100%
|
||||
app/rag/__init__.py 0 0 100%
|
||||
app/rag/agent.py 412 1 99%
|
||||
app/rag/archive_upload.py 134 0 100%
|
||||
app/rag/chunker.py 220 4 98%
|
||||
app/rag/doc_dates.py 18 0 100%
|
||||
app/rag/folder_summaries.py 134 0 100%
|
||||
app/rag/git_sources.py 38 0 100%
|
||||
app/rag/importer.py 595 9 98%
|
||||
app/rag/llm.py 246 1 99%
|
||||
app/rag/md_images.py 63 0 100%
|
||||
app/rag/overview.py 71 0 100%
|
||||
app/rag/pdf.py 109 0 100%
|
||||
app/rag/prompts.py 144 0 100%
|
||||
app/rag/retriever.py 248 3 99%
|
||||
app/rag/scaffolding.py 55 0 100%
|
||||
app/rag/source_removal.py 41 0 100%
|
||||
app/rag/sources_meta.py 16 0 100%
|
||||
app/rag/suggestions.py 27 0 100%
|
||||
app/rag/summarizer.py 49 0 100%
|
||||
app/schemas.py 410 0 100%
|
||||
--------------------------------------------------
|
||||
TOTAL 5258 22 99%
|
||||
coverage gate: app/ 99% (>90%) OK
|
||||
All checks passed!
|
||||
0 errors, 0 warnings, 0 informations
|
||||
WARNING: there is a new pyright version available (v1.1.411 -> v1.1.414).
|
||||
Please install the new version or set PYRIGHT_PYTHON_FORCE_VERSION to `latest`
|
||||
|
||||
validation OK
|
||||
+19
@@ -0,0 +1,19 @@
|
||||
All green. Task 04 is complete.
|
||||
|
||||
**Implemented**
|
||||
- `tests/e2e/test_garbage_pdf.py` (new) — the phase-133 dedicated Playwright suite, house pattern: one module app (port 8149, module env with `BOR_IMPORT_EXTENSIONS` pinned incl. `pdf`, `BOR_IMAGES=1`, leak guards, scratch image/upload homes), `kind=local` host temp dirs registered via the admin git-sources API, autouse truncate with no-running-sync wait, mock-only.
|
||||
- Fixtures: `hnb/` (real `hnb_2026_guide.tar.gz` extracted module-scoped, skips only this test if the untracked fixture is missing), `garbagedoc/` (2-page PyMuPDF garbage PDF — owner's exact `<illustrator:StartupProfile>Web</…>` line ×20/page + one 1×1 PNG embedded per page), `cleandoc/` (1-page prose PDF).
|
||||
- Tests: (1) real guide syncs `success` — content endpoint carries "2026 U.S. Benefits Guide" with **zero** XML-ish lines, and `/document.html` renders that content (stem title, page markers, `<illustrator:…>` absent); (2) garbage PDF content is byte-exactly the two pages of the mock's canned `IMAGE_DESCRIPTION_MODE` description (first-40-char pin, count, exact shape), garbage text not stored, both `#page=N#img=0.png` image docs indexed, `detail` shape unchanged (`pdfs_screenshotted` asserted absent — log-only); (3) clean PDF byte-faithful to its text layer, no fallback vision text.
|
||||
|
||||
**Results (all commands)**
|
||||
- `uv run pytest tests/e2e/test_garbage_pdf.py -v --no-cov` → **3 passed** (in isolation, DB up)
|
||||
- `uv run pytest --cov=app --cov-report=term-missing` → **3129 passed**, `app/` coverage **99%** (>90%)
|
||||
- `uv run ruff check .` → clean; `uv run pyright` → 0 errors
|
||||
|
||||
**Docs**: no doc impact (test-only task; phase-133 behavior already documented in README by tasks 01–03).
|
||||
|
||||
**Notable decisions**
|
||||
- Task mapping's `added == 1` for the garbage sync is superseded by the phase-126 A3 convention (documented in the task-01 pin's docstring): with the mandated embedded images + `BOR_IMAGES=1`, `added` counts the PDF + its 2 image docs → asserted `added == 1 + len(image_docs)` (verified live: 3), plus `files == 1` and the "invisible in the status shape" intent pinned via the absent `pdfs_screenshotted` key.
|
||||
- Verified via an instrumented probe that the real-guide sync genuinely imports (48 pages, 27 image docs, clean content, `pdfs_screenshotted=0` in the log line) — the fast runtime is the mock, not skipped work.
|
||||
|
||||
**Next pending task**: none — this was the final task (04) of phase `133_pdf_screenshot_fallback`.
|
||||
+115
@@ -0,0 +1,115 @@
|
||||
........................................................................ [ 2%]
|
||||
........................................................................ [ 4%]
|
||||
........................................................................ [ 6%]
|
||||
........................................................................ [ 9%]
|
||||
........................................................................ [ 11%]
|
||||
........................................................................ [ 13%]
|
||||
........................................................................ [ 16%]
|
||||
........................................................................ [ 18%]
|
||||
........................................................................ [ 20%]
|
||||
........................................................................ [ 23%]
|
||||
........................................................................ [ 25%]
|
||||
........................................................................ [ 27%]
|
||||
........................................................................ [ 29%]
|
||||
........................................................................ [ 32%]
|
||||
........................................................................ [ 34%]
|
||||
........................................................................ [ 36%]
|
||||
........................................................................ [ 39%]
|
||||
........................................................................ [ 41%]
|
||||
........................................................................ [ 43%]
|
||||
........................................................................ [ 46%]
|
||||
........................................................................ [ 48%]
|
||||
........................................................................ [ 50%]
|
||||
........................................................................ [ 52%]
|
||||
........................................................................ [ 55%]
|
||||
........................................................................ [ 57%]
|
||||
........................................................................ [ 59%]
|
||||
........................................................................ [ 62%]
|
||||
........................................................................ [ 64%]
|
||||
........................................................................ [ 66%]
|
||||
........................................................................ [ 69%]
|
||||
........................................................................ [ 71%]
|
||||
........................................................................ [ 73%]
|
||||
........................................................................ [ 75%]
|
||||
........................................................................ [ 78%]
|
||||
........................................................................ [ 80%]
|
||||
........................................................................ [ 82%]
|
||||
........................................................................ [ 85%]
|
||||
........................................................................ [ 87%]
|
||||
........................................................................ [ 89%]
|
||||
........................................................................ [ 92%]
|
||||
........................................................................ [ 94%]
|
||||
........................................................................ [ 96%]
|
||||
........................................................................ [ 98%]
|
||||
................................. [100%]
|
||||
=============================== warnings summary ===============================
|
||||
.venv/lib/python3.13/site-packages/fastapi/testclient.py:1
|
||||
/var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
|
||||
from starlette.testclient import TestClient as TestClient # noqa
|
||||
|
||||
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
|
||||
================================ tests coverage ================================
|
||||
_______________ coverage: platform linux, python 3.13.13-final-0 _______________
|
||||
|
||||
Name Stmts Miss Cover
|
||||
--------------------------------------------------
|
||||
app/__init__.py 1 0 100%
|
||||
app/api/__init__.py 0 0 100%
|
||||
app/api/auth.py 52 0 100%
|
||||
app/api/chat.py 268 1 99%
|
||||
app/api/chat_images.py 50 0 100%
|
||||
app/api/chats.py 110 0 100%
|
||||
app/api/config.py 13 0 100%
|
||||
app/api/doc_drafts.py 99 0 100%
|
||||
app/api/docs.py 179 1 99%
|
||||
app/api/git_sources.py 241 0 100%
|
||||
app/api/health.py 10 0 100%
|
||||
app/api/steering.py 42 0 100%
|
||||
app/api/suggestions.py 33 0 100%
|
||||
app/api/sync.py 156 0 100%
|
||||
app/api/tokens.py 40 0 100%
|
||||
app/api/ui_settings.py 55 0 100%
|
||||
app/config.py 256 0 100%
|
||||
app/core/__init__.py 0 0 100%
|
||||
app/core/auth.py 45 0 100%
|
||||
app/core/caching.py 124 0 100%
|
||||
app/core/debugging.py 29 2 93%
|
||||
app/core/docs_push.py 39 0 100%
|
||||
app/core/errors.py 5 0 100%
|
||||
app/core/logging.py 13 0 100%
|
||||
app/core/rate_limit.py 44 0 100%
|
||||
app/core/security_headers.py 20 0 100%
|
||||
app/core/theming.py 38 0 100%
|
||||
app/core/tokens.py 44 0 100%
|
||||
app/db.py 22 0 100%
|
||||
app/main.py 68 0 100%
|
||||
app/models.py 132 0 100%
|
||||
app/rag/__init__.py 0 0 100%
|
||||
app/rag/agent.py 412 1 99%
|
||||
app/rag/archive_upload.py 134 0 100%
|
||||
app/rag/chunker.py 220 4 98%
|
||||
app/rag/doc_dates.py 18 0 100%
|
||||
app/rag/folder_summaries.py 134 0 100%
|
||||
app/rag/git_sources.py 38 0 100%
|
||||
app/rag/importer.py 595 9 98%
|
||||
app/rag/llm.py 246 1 99%
|
||||
app/rag/md_images.py 63 0 100%
|
||||
app/rag/overview.py 71 0 100%
|
||||
app/rag/pdf.py 109 0 100%
|
||||
app/rag/prompts.py 144 0 100%
|
||||
app/rag/retriever.py 248 3 99%
|
||||
app/rag/scaffolding.py 55 0 100%
|
||||
app/rag/source_removal.py 41 0 100%
|
||||
app/rag/sources_meta.py 16 0 100%
|
||||
app/rag/suggestions.py 27 0 100%
|
||||
app/rag/summarizer.py 49 0 100%
|
||||
app/schemas.py 410 0 100%
|
||||
--------------------------------------------------
|
||||
TOTAL 5258 22 99%
|
||||
coverage gate: app/ 99% (>90%) OK
|
||||
All checks passed!
|
||||
0 errors, 0 warnings, 0 informations
|
||||
WARNING: there is a new pyright version available (v1.1.411 -> v1.1.414).
|
||||
Please install the new version or set PYRIGHT_PYTHON_FORCE_VERSION to `latest`
|
||||
|
||||
validation OK
|
||||
Reference in New Issue
Block a user