Experimental/brain-of-reese
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

phase: 137_pdf_image_context
Build and Push Containers / build-and-push-app (push) Successful in 2m51s
Details
Build and Push Containers / build-and-push-db (push) Successful in 13s
Details

Browse Source 
All verification complete — everything is 100% green. Final report:

**Phase 137 final verification pass — all tasks already complete, all criteria verified**

- **Full suite:** `uv run pytest --cov=app --cov-report=term-missing` → 3272 passed, exit 0; `app/` coverage **99%** (>90% ✓)
- **Phase E2E in isolation:** `uv run pytest tests/e2e/test_pdf_image_context.py -v --no-cov` → **4 passed** (forward context, display line, standalone parity, both-blocks markdown case)
- **Pre-existing suites in isolation:** `test_pdf_support.py` + `test_image_anchor_context.py` + `test_image_documents.py` → **14 passed**; ripple-edited `test_garbage_pdf.py` → **3 passed**
- **Lint/types:** `uv run ruff check .` → all passed; `uv run pyright` → 0 errors

**Completion criteria:** (1) forward-path context in vision msg + stored `image_context`/summary — ✓ (unit builder/seam + integration fake-LLM + E2E); (2) both blocks document-first + standalone describe byte-identical — ✓ (combiner + standalone pins); (3) backfill with context + zero-vision idempotent re-sync + parent-gone origin-only — ✓ (6 integration tests); (4) suggestion block leads with `From <source>/<pdf> (page N)` — ✓ (renderer units + E2E); (5) all gates green — ✓ above; (6) no behavior change in phases 122/125/126 + sync `detail` shape untouched — ✓ (verified in code: new counter is log-only); (7) ruff+pyright — ✓; (8) commit — left to the harness per pipeline rules (working tree untouched by me).

**Documentation:** README.md audited in full — the new phase-137 bullet verified against code (6 000-char cap, log-only `image_context_backfilled=N`, `<document>`-body locator placement), all env tables/defaults cross-checked vs `app/config.py` and `.env.example` — consistent, no changes needed. AGENTS.md: every rule/command verified — consistent, no changes needed.

**Notable deviations (pre-existing, documented in task-05 report, forced by design):** `test_pdf_support.py`/`test_garbage_pdf.py` got minimal ripple edits (the "UNEDITED" pin was unsatisfiable once describe calls carry context by design); the unit `test_image_anchor_context.py` CONTEXT_PREFIX pin generalized per ASSUMPTION A5; one extra mock line (`_seed_context_line` accepts `From …`).

**Next pending phase:** none — `todo/` holds only this phase.
This commit is contained in:
ducoterra
2026-09-29 02:14:46 -04:00
parent 0035bdf533
commit 5e87622b30
39 changed files with 3565 additions and 167 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.agents/phases/todo/137_pdf_image_context/00_phase.md → .agents/phases/complete/137_pdf_image_context/00_phase.md
View File
.agents/phases/todo/137_pdf_image_context/01_pdf_page_texts.md → .agents/phases/complete/137_pdf_image_context/01_pdf_page_texts.md
View File
.agents/phases/todo/137_pdf_image_context/02_embedded_image_document_context.md → .agents/phases/complete/137_pdf_image_context/02_embedded_image_document_context.md
View File
.agents/phases/todo/137_pdf_image_context/03_backfill_image_context.md → .agents/phases/complete/137_pdf_image_context/03_backfill_image_context.md
View File
.agents/phases/todo/137_pdf_image_context/04_retrieval_display_line.md → .agents/phases/complete/137_pdf_image_context/04_retrieval_display_line.md
View File
.agents/phases/todo/137_pdf_image_context/05_e2e_pdf_image_context.md → .agents/phases/complete/137_pdf_image_context/05_e2e_pdf_image_context.md
View File
.agents/reports/137_pdf_image_context/137_pdf_image_context__00_phase.a1.err
View File
.agents/reports/137_pdf_image_context/137_pdf_image_context__00_phase.a1.md
+16
View File
@@ -0,0 +1,16 @@
All verification complete — everything is 100% green. Final report:
**Phase 137 final verification pass — all tasks already complete, all criteria verified**
- **Full suite:** `uv run pytest --cov=app --cov-report=term-missing` → 3272 passed, exit 0; `app/` coverage **99%** (>90% ✓)
- **Phase E2E in isolation:** `uv run pytest tests/e2e/test_pdf_image_context.py -v --no-cov` → **4 passed** (forward context, display line, standalone parity, both-blocks markdown case)
- **Pre-existing suites in isolation:** `test_pdf_support.py` + `test_image_anchor_context.py` + `test_image_documents.py` → **14 passed**; ripple-edited `test_garbage_pdf.py` → **3 passed**
- **Lint/types:** `uv run ruff check .` → all passed; `uv run pyright` → 0 errors
**Completion criteria:** (1) forward-path context in vision msg + stored `image_context`/summary — ✓ (unit builder/seam + integration fake-LLM + E2E); (2) both blocks document-first + standalone describe byte-identical — ✓ (combiner + standalone pins); (3) backfill with context + zero-vision idempotent re-sync + parent-gone origin-only — ✓ (6 integration tests); (4) suggestion block leads with `From <source>/<pdf> (page N)` — ✓ (renderer units + E2E); (5) all gates green — ✓ above; (6) no behavior change in phases 122/125/126 + sync `detail` shape untouched — ✓ (verified in code: new counter is log-only); (7) ruff+pyright — ✓; (8) commit — left to the harness per pipeline rules (working tree untouched by me).
**Documentation:** README.md audited in full — the new phase-137 bullet verified against code (6 000-char cap, log-only `image_context_backfilled=N`, `<document>`-body locator placement), all env tables/defaults cross-checked vs `app/config.py` and `.env.example` — consistent, no changes needed. AGENTS.md: every rule/command verified — consistent, no changes needed.
**Notable deviations (pre-existing, documented in task-05 report, forced by design):** `test_pdf_support.py`/`test_garbage_pdf.py` got minimal ripple edits (the "UNEDITED" pin was unsatisfiable once describe calls carry context by design); the unit `test_image_anchor_context.py` CONTEXT_PREFIX pin generalized per ASSUMPTION A5; one extra mock line (`_seed_context_line` accepts `From …`).
**Next pending phase:** none — `todo/` holds only this phase.
.agents/reports/137_pdf_image_context/137_pdf_image_context__00_phase.a1.validate
+117
View File
@@ -0,0 +1,117 @@
........................................................................ [ 2%]
........................................................................ [ 4%]
........................................................................ [ 6%]
........................................................................ [ 8%]
........................................................................ [ 11%]
........................................................................ [ 13%]
........................................................................ [ 15%]
........................................................................ [ 17%]
........................................................................ [ 19%]
........................................................................ [ 22%]
........................................................................ [ 24%]
........................................................................ [ 26%]
........................................................................ [ 28%]
........................................................................ [ 30%]
........................................................................ [ 33%]
........................................................................ [ 35%]
........................................................................ [ 37%]
........................................................................ [ 39%]
........................................................................ [ 41%]
........................................................................ [ 44%]
........................................................................ [ 46%]
........................................................................ [ 48%]
........................................................................ [ 50%]
........................................................................ [ 52%]
........................................................................ [ 55%]
........................................................................ [ 57%]
........................................................................ [ 59%]
........................................................................ [ 61%]
........................................................................ [ 63%]
........................................................................ [ 66%]
........................................................................ [ 68%]
........................................................................ [ 70%]
........................................................................ [ 72%]
........................................................................ [ 74%]
........................................................................ [ 77%]
........................................................................ [ 79%]
........................................................................ [ 81%]
........................................................................ [ 83%]
........................................................................ [ 85%]
........................................................................ [ 88%]
........................................................................ [ 90%]
........................................................................ [ 92%]
........................................................................ [ 94%]
........................................................................ [ 96%]
........................................................................ [ 99%]
................................ [100%]
=============================== warnings summary ===============================
.venv/lib/python3.13/site-packages/fastapi/testclient.py:1
/var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
from starlette.testclient import TestClient as TestClient # noqa
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
================================ tests coverage ================================
_______________ coverage: platform linux, python 3.13.13-final-0 _______________
Name Stmts Miss Cover
--------------------------------------------------
app/__init__.py 1 0 100%
app/api/__init__.py 0 0 100%
app/api/auth.py 52 0 100%
app/api/chat.py 268 1 99%
app/api/chat_images.py 50 0 100%
app/api/chats.py 110 0 100%
app/api/config.py 13 0 100%
app/api/doc_drafts.py 99 0 100%
app/api/docs.py 179 1 99%
app/api/git_sources.py 259 0 100%
app/api/health.py 10 0 100%
app/api/steering.py 42 0 100%
app/api/suggestions.py 33 0 100%
app/api/sync.py 183 0 100%
app/api/tokens.py 40 0 100%
app/api/ui_settings.py 55 0 100%
app/config.py 256 0 100%
app/core/__init__.py 0 0 100%
app/core/auth.py 45 0 100%
app/core/caching.py 124 0 100%
app/core/debugging.py 29 2 93%
app/core/docs_push.py 39 0 100%
app/core/errors.py 5 0 100%
app/core/logging.py 13 0 100%
app/core/rate_limit.py 44 0 100%
app/core/security_headers.py 20 0 100%
app/core/theming.py 38 0 100%
app/core/tokens.py 44 0 100%
app/db.py 22 0 100%
app/main.py 68 0 100%
app/models.py 133 0 100%
app/rag/__init__.py 0 0 100%
app/rag/agent.py 412 1 99%
app/rag/archive_upload.py 134 0 100%
app/rag/chunker.py 220 4 98%
app/rag/doc_dates.py 18 0 100%
app/rag/folder_summaries.py 134 0 100%
app/rag/git_sources.py 56 0 100%
app/rag/importer.py 671 9 99%
app/rag/llm.py 246 1 99%
app/rag/md_images.py 63 0 100%
app/rag/overview.py 71 0 100%
app/rag/pdf.py 119 0 100%
app/rag/prompts.py 154 0 100%
app/rag/retriever.py 248 3 99%
app/rag/scaffolding.py 55 0 100%
app/rag/source_removal.py 41 0 100%
app/rag/sources_meta.py 16 0 100%
app/rag/suggestions.py 27 0 100%
app/rag/summarizer.py 49 0 100%
app/schemas.py 422 0 100%
--------------------------------------------------
TOTAL 5430 22 99%
coverage gate: app/ 99% (>90%) OK
All checks passed!
0 errors, 0 warnings, 0 informations
WARNING: there is a new pyright version available (v1.1.411 -> v1.1.414).
Please install the new version or set PYRIGHT_PYTHON_FORCE_VERSION to `latest`
validation OK
.agents/reports/137_pdf_image_context/137_pdf_image_context__01_pdf_page_texts.a1.err
View File
.agents/reports/137_pdf_image_context/137_pdf_image_context__01_pdf_page_texts.a1.md
+11
View File
@@ -0,0 +1,11 @@
Everything is verified. The full coverage run (3239 passed) happened after the final code edit, and ruff/pyright ran clean after it too — no changes since.
**Report — Task 01: `extract_pdf_page_texts` (phase 137)**
- `app/rag/pdf.py`: new private `_iter_page_texts(doc, pdf_path)` — the shared per-page iteration (per-page try/except + warning logging moved in, unchanged); `extract_pdf_text` now consumes it (signature/output/log lines byte-identical); new public fail-soft `extract_pdf_page_texts(pdf_path) -> dict[int, str]` (unreadable file → `{}` + one `pdf:`-prefixed warning); module docstring function list updated.
- `tests/unit/test_pdf.py`: 8 new tests — handbook (291/293 entries; exact page-for-page equality with parsed `---PAGE-N---` blocks; 1-indexed keys, all values stripped; spot-checks of known page bodies), horse PDF (exactly 1 entry, asserted against `extract_pdf_text` whatever it is), missing/garbage path → `{}`, broken-page skip through the new call site, and the pre-refactor byte-identity pins (length + sha256 + separator structure for the 293-page handbook; exact string pin for the horse) captured from the pre-refactor output.
- Tests: `uv run pytest tests/unit -k pdf -v --no-cov` → 89 passed. Full suite `uv run pytest --cov=app --cov-report=term-missing` → **3239 passed, TOTAL 99%**, `app/rag/pdf.py` 100%.
- Lint/types: `uv run ruff check .` → All checks passed; `uv run pyright` → 0 errors, 0 warnings.
- Docs: no doc impact — internal parser helper; no env/commands/API/convention changes; README/AGENTS.md don't reference `app/rag/pdf` internals.
- Notable deviation: the task says "first key `1`" for the handbook, but its cover (page 1) and page 280 are image-only (verified against the fixture), so the first key is 2 — pinned with an explanatory comment; 1-indexedness is proven via page-for-page agreement with `extract_pdf_text`, and the horse fixture does pin first key `1`.
- Next pending task: `02_embedded_image_document_context.md` (same phase).
.agents/reports/137_pdf_image_context/137_pdf_image_context__01_pdf_page_texts.a1.validate
+116
View File
@@ -0,0 +1,116 @@
........................................................................ [ 2%]
........................................................................ [ 4%]
........................................................................ [ 6%]
........................................................................ [ 8%]
........................................................................ [ 11%]
........................................................................ [ 13%]
........................................................................ [ 15%]
........................................................................ [ 17%]
........................................................................ [ 20%]
........................................................................ [ 22%]
........................................................................ [ 24%]
........................................................................ [ 26%]
........................................................................ [ 28%]
........................................................................ [ 31%]
........................................................................ [ 33%]
........................................................................ [ 35%]
........................................................................ [ 37%]
........................................................................ [ 40%]
........................................................................ [ 42%]
........................................................................ [ 44%]
........................................................................ [ 46%]
........................................................................ [ 48%]
........................................................................ [ 51%]
........................................................................ [ 53%]
........................................................................ [ 55%]
........................................................................ [ 57%]
........................................................................ [ 60%]
........................................................................ [ 62%]
........................................................................ [ 64%]
........................................................................ [ 66%]
........................................................................ [ 68%]
........................................................................ [ 71%]
........................................................................ [ 73%]
........................................................................ [ 75%]
........................................................................ [ 77%]
........................................................................ [ 80%]
........................................................................ [ 82%]
........................................................................ [ 84%]
........................................................................ [ 86%]
........................................................................ [ 88%]
........................................................................ [ 91%]
........................................................................ [ 93%]
........................................................................ [ 95%]
........................................................................ [ 97%]
....................................................................... [100%]
=============================== warnings summary ===============================
.venv/lib/python3.13/site-packages/fastapi/testclient.py:1
/var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
from starlette.testclient import TestClient as TestClient # noqa
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
================================ tests coverage ================================
_______________ coverage: platform linux, python 3.13.13-final-0 _______________
Name Stmts Miss Cover
--------------------------------------------------
app/__init__.py 1 0 100%
app/api/__init__.py 0 0 100%
app/api/auth.py 52 0 100%
app/api/chat.py 268 1 99%
app/api/chat_images.py 50 0 100%
app/api/chats.py 110 0 100%
app/api/config.py 13 0 100%
app/api/doc_drafts.py 99 0 100%
app/api/docs.py 179 1 99%
app/api/git_sources.py 259 0 100%
app/api/health.py 10 0 100%
app/api/steering.py 42 0 100%
app/api/suggestions.py 33 0 100%
app/api/sync.py 183 0 100%
app/api/tokens.py 40 0 100%
app/api/ui_settings.py 55 0 100%
app/config.py 256 0 100%
app/core/__init__.py 0 0 100%
app/core/auth.py 45 0 100%
app/core/caching.py 124 0 100%
app/core/debugging.py 29 2 93%
app/core/docs_push.py 39 0 100%
app/core/errors.py 5 0 100%
app/core/logging.py 13 0 100%
app/core/rate_limit.py 44 0 100%
app/core/security_headers.py 20 0 100%
app/core/theming.py 38 0 100%
app/core/tokens.py 44 0 100%
app/db.py 22 0 100%
app/main.py 68 0 100%
app/models.py 133 0 100%
app/rag/__init__.py 0 0 100%
app/rag/agent.py 412 1 99%
app/rag/archive_upload.py 134 0 100%
app/rag/chunker.py 220 4 98%
app/rag/doc_dates.py 18 0 100%
app/rag/folder_summaries.py 134 0 100%
app/rag/git_sources.py 56 0 100%
app/rag/importer.py 595 9 98%
app/rag/llm.py 246 1 99%
app/rag/md_images.py 63 0 100%
app/rag/overview.py 71 0 100%
app/rag/pdf.py 119 0 100%
app/rag/prompts.py 144 0 100%
app/rag/retriever.py 248 3 99%
app/rag/scaffolding.py 55 0 100%
app/rag/source_removal.py 41 0 100%
app/rag/sources_meta.py 16 0 100%
app/rag/suggestions.py 27 0 100%
app/rag/summarizer.py 49 0 100%
app/schemas.py 422 0 100%
--------------------------------------------------
TOTAL 5344 22 99%
coverage gate: app/ 99% (>90%) OK
All checks passed!
0 errors, 0 warnings, 0 informations
WARNING: there is a new pyright version available (v1.1.411 -> v1.1.414).
Please install the new version or set PYRIGHT_PYTHON_FORCE_VERSION to `latest`
validation OK
.agents/reports/137_pdf_image_context/137_pdf_image_context__02_embedded_image_document_context.a1.err
View File
.agents/reports/137_pdf_image_context/137_pdf_image_context__02_embedded_image_document_context.a1.md
+12
View File
@@ -0,0 +1,12 @@
All green. Task 02 is complete.
**Report — phase 137, task 02 (embedded-image document context, forward path)**
- `app/rag/importer.py`: `PDF_IMAGE_CONTEXT_MAX_CHARS=6000`, `_pdf_image_context_block` (origin line always; page text stripped, capped + `TRUNCATION_MARKER`; origin-only for text-less page or gate-flagged layer), `_combine_image_contexts` (doc block first — shared with task 03); `_index_pdf_embedded_images` reads `extract_pdf_page_texts` once per PDF + runs the phase-133 gate → `text_usable`; combined context flows to the vision seam, stored `image_context`, and `content`/`summary`.
- `app/rag/summarizer.py`: `describe_image` context prefix generalized to neutral wording (A5); `DESCRIBE_PROMPT` byte-identical; `context=None` sends the bare prompt (pinned).
- Tests: 10 new unit tests in `tests/unit/test_pdf.py` (builder cap/edges, combining exact strings, capturing-LLM forward path, both-blocks, text-less page, garbage-layer origin-only, standalone byte-identity, unchanged-skip) + updated pins in `test_image_anchor_context.py`, `tests/integration/test_importer.py` (3 PDF tests + prefix constant), `test_pdf_hnb_guide_regression.py` (586-photo guide now asserted with per-page context blocks).
- E2E: `tests/e2e/test_pdf_support.py` `seeded` fixture + `tests/e2e/test_garbage_pdf.py` updated to the new content shape (the old `content == ANSWER` pins encode pre-phase behavior this task intentionally changes — see deviations).
- Results: `uv run pytest --cov=app` → **3251 passed, TOTAL 99%** (>90%); `uv run ruff check . && uv run pyright` → clean; E2E in isolation: `test_pdf_support.py` 6✓, `test_garbage_pdf.py` 3✓, `test_image_anchor_context.py` 4✓, `test_image_documents.py` 4✓, `test_answer_images.py` 5✓, `test_chat_answer_clusters.py` 4✓.
- Docs: README PDF section gained a "Document context (phase 137)" bullet; AGENTS.md no impact.
- Deviations: (1) used the phase-133 gate directly — 133 is already complete and the gate is an existing module-level import (the task's defensive-import note predates its landing); (2) `test_pdf_support.py` needed minimal edits despite the "unedited" criterion — its content asserts are incompatible with the phase's own objective.
- Next pending task: `03_backfill_image_context.md`.
.agents/reports/137_pdf_image_context/137_pdf_image_context__02_embedded_image_document_context.a1.validate
+117
View File
@@ -0,0 +1,117 @@
........................................................................ [ 2%]
........................................................................ [ 4%]
........................................................................ [ 6%]
........................................................................ [ 8%]
........................................................................ [ 11%]
........................................................................ [ 13%]
........................................................................ [ 15%]
........................................................................ [ 17%]
........................................................................ [ 19%]
........................................................................ [ 22%]
........................................................................ [ 24%]
........................................................................ [ 26%]
........................................................................ [ 28%]
........................................................................ [ 31%]
........................................................................ [ 33%]
........................................................................ [ 35%]
........................................................................ [ 37%]
........................................................................ [ 39%]
........................................................................ [ 42%]
........................................................................ [ 44%]
........................................................................ [ 46%]
........................................................................ [ 48%]
........................................................................ [ 50%]
........................................................................ [ 53%]
........................................................................ [ 55%]
........................................................................ [ 57%]
........................................................................ [ 59%]
........................................................................ [ 62%]
........................................................................ [ 64%]
........................................................................ [ 66%]
........................................................................ [ 68%]
........................................................................ [ 70%]
........................................................................ [ 73%]
........................................................................ [ 75%]
........................................................................ [ 77%]
........................................................................ [ 79%]
........................................................................ [ 81%]
........................................................................ [ 84%]
........................................................................ [ 86%]
........................................................................ [ 88%]
........................................................................ [ 90%]
........................................................................ [ 93%]
........................................................................ [ 95%]
........................................................................ [ 97%]
........................................................................ [ 99%]
........... [100%]
=============================== warnings summary ===============================
.venv/lib/python3.13/site-packages/fastapi/testclient.py:1
/var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
from starlette.testclient import TestClient as TestClient # noqa
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
================================ tests coverage ================================
_______________ coverage: platform linux, python 3.13.13-final-0 _______________
Name Stmts Miss Cover
--------------------------------------------------
app/__init__.py 1 0 100%
app/api/__init__.py 0 0 100%
app/api/auth.py 52 0 100%
app/api/chat.py 268 1 99%
app/api/chat_images.py 50 0 100%
app/api/chats.py 110 0 100%
app/api/config.py 13 0 100%
app/api/doc_drafts.py 99 0 100%
app/api/docs.py 179 1 99%
app/api/git_sources.py 259 0 100%
app/api/health.py 10 0 100%
app/api/steering.py 42 0 100%
app/api/suggestions.py 33 0 100%
app/api/sync.py 183 0 100%
app/api/tokens.py 40 0 100%
app/api/ui_settings.py 55 0 100%
app/config.py 256 0 100%
app/core/__init__.py 0 0 100%
app/core/auth.py 45 0 100%
app/core/caching.py 124 0 100%
app/core/debugging.py 29 2 93%
app/core/docs_push.py 39 0 100%
app/core/errors.py 5 0 100%
app/core/logging.py 13 0 100%
app/core/rate_limit.py 44 0 100%
app/core/security_headers.py 20 0 100%
app/core/theming.py 38 0 100%
app/core/tokens.py 44 0 100%
app/db.py 22 0 100%
app/main.py 68 0 100%
app/models.py 133 0 100%
app/rag/__init__.py 0 0 100%
app/rag/agent.py 412 1 99%
app/rag/archive_upload.py 134 0 100%
app/rag/chunker.py 220 4 98%
app/rag/doc_dates.py 18 0 100%
app/rag/folder_summaries.py 134 0 100%
app/rag/git_sources.py 56 0 100%
app/rag/importer.py 612 9 99%
app/rag/llm.py 246 1 99%
app/rag/md_images.py 63 0 100%
app/rag/overview.py 71 0 100%
app/rag/pdf.py 119 0 100%
app/rag/prompts.py 144 0 100%
app/rag/retriever.py 248 3 99%
app/rag/scaffolding.py 55 0 100%
app/rag/source_removal.py 41 0 100%
app/rag/sources_meta.py 16 0 100%
app/rag/suggestions.py 27 0 100%
app/rag/summarizer.py 49 0 100%
app/schemas.py 422 0 100%
--------------------------------------------------
TOTAL 5361 22 99%
coverage gate: app/ 99% (>90%) OK
All checks passed!
0 errors, 0 warnings, 0 informations
WARNING: there is a new pyright version available (v1.1.411 -> v1.1.414).
Please install the new version or set PYRIGHT_PYTHON_FORCE_VERSION to `latest`
validation OK
.agents/reports/137_pdf_image_context/137_pdf_image_context__03_backfill_image_context.a1.err
View File
.agents/reports/137_pdf_image_context/137_pdf_image_context__03_backfill_image_context.a1.md
+10
View File
@@ -0,0 +1,10 @@
Everything is green. Final report:
**Task 03 — backfill image context (phase 137): complete**
- **Implemented** (`app/rag/importer.py`): `_backfill_image_contexts` — per-sync, idempotent, fail-soft pass run in `import_sources` immediately after `_enrich_pending_images`; candidates = `is_image` rows matching the phase-126 synthetic-path regex (new `_EMBEDDED_IMAGE_PATH_RE`, PDF suffix case-insensitive to match the walk) with `image_context` NULL/markdown-only and an existing `image_dir` copy; per candidate: parent page text re-extracted once per unique `pdf_rel` via new shared `_pdf_image_context_inputs` probe (parent gone → origin line only), combined with stored markdown blocks verbatim (document first) through the task-02 helpers, re-described through the same `_describe_or_skip` seam, row's `content`/`summary`/`image_context`/chunks replaced in its own transaction (`content_hash`/dates/`title`/`image_path` untouched); `ImportSummary.image_context_backfilled` log-only counter + `log()` field next to `images_failed`.
- **Tests**: 9 new integration tests in `tests/integration/test_importer.py` (real `import_sources`, fake LLM, seeded rows): happy path, idempotency (zero vision calls + row byte-identical), parent-gone origin-only, markdown-block-verbatim, standalone/copyless never touched (two runs), unreadable-copy skip, describe-fail retry, embed-fail rollback+retry (copy stays), uppercase `.PDF` extension.
- **Results**: `uv run pytest tests/integration -k "image or import" --no-cov` → 136 passed; `uv run pytest --cov=app --cov-report=term-missing` → **3260 passed, app/ 99%** (importer.py 99%, no new uncovered lines); `uv run ruff check .` clean; `uv run pyright` 0 errors. Regression: `test_sync_api.py` (sync `detail` shape) unedited 29 passed; E2E `test_pdf_support`/`test_image_documents`/`test_image_anchor_context`/`test_garbage_pdf` + all 8 sync E2E files green unedited.
- **Docs**: README phase-137 bullet extended with the backfill behavior (log-only counter); AGENTS.md no impact.
- **Decisions**: backfill success counts only `image_context_backfilled` (no `updated`/`changed_paths` — literal task spec; sync detail/version-bump gates untouched); task-02's inline gate probe factored into the shared `_pdf_image_context_inputs` helper (behavior-identical) so forward/backfill probes can't drift; pinned the updated `ImportSummary.log()` line in `tests/unit/test_importer.py`.
- **Next pending task**: `04_retrieval_display_line.md` (same phase).
.agents/reports/137_pdf_image_context/137_pdf_image_context__03_backfill_image_context.a1.validate
+117
View File
@@ -0,0 +1,117 @@
........................................................................ [ 2%]
........................................................................ [ 4%]
........................................................................ [ 6%]
........................................................................ [ 8%]
........................................................................ [ 11%]
........................................................................ [ 13%]
........................................................................ [ 15%]
........................................................................ [ 17%]
........................................................................ [ 19%]
........................................................................ [ 22%]
........................................................................ [ 24%]
........................................................................ [ 26%]
........................................................................ [ 28%]
........................................................................ [ 30%]
........................................................................ [ 33%]
........................................................................ [ 35%]
........................................................................ [ 37%]
........................................................................ [ 39%]
........................................................................ [ 41%]
........................................................................ [ 44%]
........................................................................ [ 46%]
........................................................................ [ 48%]
........................................................................ [ 50%]
........................................................................ [ 53%]
........................................................................ [ 55%]
........................................................................ [ 57%]
........................................................................ [ 59%]
........................................................................ [ 61%]
........................................................................ [ 64%]
........................................................................ [ 66%]
........................................................................ [ 68%]
........................................................................ [ 70%]
........................................................................ [ 72%]
........................................................................ [ 75%]
........................................................................ [ 77%]
........................................................................ [ 79%]
........................................................................ [ 81%]
........................................................................ [ 83%]
........................................................................ [ 86%]
........................................................................ [ 88%]
........................................................................ [ 90%]
........................................................................ [ 92%]
........................................................................ [ 94%]
........................................................................ [ 97%]
........................................................................ [ 99%]
.................... [100%]
=============================== warnings summary ===============================
.venv/lib/python3.13/site-packages/fastapi/testclient.py:1
/var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
from starlette.testclient import TestClient as TestClient # noqa
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
================================ tests coverage ================================
_______________ coverage: platform linux, python 3.13.13-final-0 _______________
Name Stmts Miss Cover
--------------------------------------------------
app/__init__.py 1 0 100%
app/api/__init__.py 0 0 100%
app/api/auth.py 52 0 100%
app/api/chat.py 268 1 99%
app/api/chat_images.py 50 0 100%
app/api/chats.py 110 0 100%
app/api/config.py 13 0 100%
app/api/doc_drafts.py 99 0 100%
app/api/docs.py 179 1 99%
app/api/git_sources.py 259 0 100%
app/api/health.py 10 0 100%
app/api/steering.py 42 0 100%
app/api/suggestions.py 33 0 100%
app/api/sync.py 183 0 100%
app/api/tokens.py 40 0 100%
app/api/ui_settings.py 55 0 100%
app/config.py 256 0 100%
app/core/__init__.py 0 0 100%
app/core/auth.py 45 0 100%
app/core/caching.py 124 0 100%
app/core/debugging.py 29 2 93%
app/core/docs_push.py 39 0 100%
app/core/errors.py 5 0 100%
app/core/logging.py 13 0 100%
app/core/rate_limit.py 44 0 100%
app/core/security_headers.py 20 0 100%
app/core/theming.py 38 0 100%
app/core/tokens.py 44 0 100%
app/db.py 22 0 100%
app/main.py 68 0 100%
app/models.py 133 0 100%
app/rag/__init__.py 0 0 100%
app/rag/agent.py 412 1 99%
app/rag/archive_upload.py 134 0 100%
app/rag/chunker.py 220 4 98%
app/rag/doc_dates.py 18 0 100%
app/rag/folder_summaries.py 134 0 100%
app/rag/git_sources.py 56 0 100%
app/rag/importer.py 671 9 99%
app/rag/llm.py 246 1 99%
app/rag/md_images.py 63 0 100%
app/rag/overview.py 71 0 100%
app/rag/pdf.py 119 0 100%
app/rag/prompts.py 144 0 100%
app/rag/retriever.py 248 3 99%
app/rag/scaffolding.py 55 0 100%
app/rag/source_removal.py 41 0 100%
app/rag/sources_meta.py 16 0 100%
app/rag/suggestions.py 27 0 100%
app/rag/summarizer.py 49 0 100%
app/schemas.py 422 0 100%
--------------------------------------------------
TOTAL 5420 22 99%
coverage gate: app/ 99% (>90%) OK
All checks passed!
0 errors, 0 warnings, 0 informations
WARNING: there is a new pyright version available (v1.1.411 -> v1.1.414).
Please install the new version or set PYRIGHT_PYTHON_FORCE_VERSION to `latest`
validation OK
.agents/reports/137_pdf_image_context/137_pdf_image_context__04_retrieval_display_line.a1.err
View File
.agents/reports/137_pdf_image_context/137_pdf_image_context__04_retrieval_display_line.a1.md
+17
View File
@@ -0,0 +1,17 @@
Task 04 complete. Report:
**Implemented**
- `app/rag/prompts.py`: added `_PDF_CTX_HEADER_RE` (`^Extracted from (?P<loc>.+) \(page (?P<page>\d+)\):$`, greedy `loc`) next to the phase-125 regex + phase-137 comment; restructured `_extract_context_line`'s state machine — an `Extracted from` header needs no bullet (clause = `From <loc> (page N)`), flushes a complete pending reference block in stored order, drops a pending one via the existing spurious-header rule; join is `"; "` in stored order with per-clause prefixes in mixed values; markdown-only values keep the single-prefix shape byte-identically (phase-125 pin); updated the docstring (both block kinds, the accepted unguarded spurious-`Extracted from`-in-page-text asymmetry) and the `_document_body` docstring example line.
- Verified `_image_context_line`/`_document_body` need NO code changes (the `is_image` guard + renderer call already do the right thing).
- `tests/unit/test_prompts.py`: +12 tests, 0 lines removed (phase-125 pins unedited): pdf-alone, text-less header-only, greedy locator with spaces, page-text body + spurious `Referenced in` line skipped, spurious `Extracted from` in page text emits a clause (accepted, pinned), markdown-only 2-block pin (`Referenced in docs/a.md: "alt a"; docs/b.md`), both-blocks document-first, markdown-first stored order, pdf garbage → `None`, `_document_body` combined + text-doc guard, built-prompt block shape.
**Results**
- `uv run pytest --cov=app --cov-report=term-missing`: 3272 passed, `app/` 99% (`prompts.py` 100%)
- `uv run pytest tests/unit -k prompt --no-cov`: 148 passed
- `uv run ruff check .` + `uv run pyright`: clean (0 errors)
**Docs**: README's phase-137 bullet extended with the visible display-line contract; AGENTS.md no impact.
**Decisions**: markdown-only values render exactly pre-task (ONE `Referenced in` prefix — the unedited phase-125 pin); mixed values give every clause its own prefix (`From …; Referenced in …`).
**Next pending task**: `05_e2e_pdf_image_context.md` (mock `Extracted from` trigger + dedicated Playwright suite).
.agents/reports/137_pdf_image_context/137_pdf_image_context__04_retrieval_display_line.a1.validate
+117
View File
@@ -0,0 +1,117 @@
........................................................................ [ 2%]
........................................................................ [ 4%]
........................................................................ [ 6%]
........................................................................ [ 8%]
........................................................................ [ 11%]
........................................................................ [ 13%]
........................................................................ [ 15%]
........................................................................ [ 17%]
........................................................................ [ 19%]
........................................................................ [ 22%]
........................................................................ [ 24%]
........................................................................ [ 26%]
........................................................................ [ 28%]
........................................................................ [ 30%]
........................................................................ [ 33%]
........................................................................ [ 35%]
........................................................................ [ 37%]
........................................................................ [ 39%]
........................................................................ [ 41%]
........................................................................ [ 44%]
........................................................................ [ 46%]
........................................................................ [ 48%]
........................................................................ [ 50%]
........................................................................ [ 52%]
........................................................................ [ 55%]
........................................................................ [ 57%]
........................................................................ [ 59%]
........................................................................ [ 61%]
........................................................................ [ 63%]
........................................................................ [ 66%]
........................................................................ [ 68%]
........................................................................ [ 70%]
........................................................................ [ 72%]
........................................................................ [ 74%]
........................................................................ [ 77%]
........................................................................ [ 79%]
........................................................................ [ 81%]
........................................................................ [ 83%]
........................................................................ [ 85%]
........................................................................ [ 88%]
........................................................................ [ 90%]
........................................................................ [ 92%]
........................................................................ [ 94%]
........................................................................ [ 96%]
........................................................................ [ 99%]
................................ [100%]
=============================== warnings summary ===============================
.venv/lib/python3.13/site-packages/fastapi/testclient.py:1
/var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
from starlette.testclient import TestClient as TestClient # noqa
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
================================ tests coverage ================================
_______________ coverage: platform linux, python 3.13.13-final-0 _______________
Name Stmts Miss Cover
--------------------------------------------------
app/__init__.py 1 0 100%
app/api/__init__.py 0 0 100%
app/api/auth.py 52 0 100%
app/api/chat.py 268 1 99%
app/api/chat_images.py 50 0 100%
app/api/chats.py 110 0 100%
app/api/config.py 13 0 100%
app/api/doc_drafts.py 99 0 100%
app/api/docs.py 179 1 99%
app/api/git_sources.py 259 0 100%
app/api/health.py 10 0 100%
app/api/steering.py 42 0 100%
app/api/suggestions.py 33 0 100%
app/api/sync.py 183 0 100%
app/api/tokens.py 40 0 100%
app/api/ui_settings.py 55 0 100%
app/config.py 256 0 100%
app/core/__init__.py 0 0 100%
app/core/auth.py 45 0 100%
app/core/caching.py 124 0 100%
app/core/debugging.py 29 2 93%
app/core/docs_push.py 39 0 100%
app/core/errors.py 5 0 100%
app/core/logging.py 13 0 100%
app/core/rate_limit.py 44 0 100%
app/core/security_headers.py 20 0 100%
app/core/theming.py 38 0 100%
app/core/tokens.py 44 0 100%
app/db.py 22 0 100%
app/main.py 68 0 100%
app/models.py 133 0 100%
app/rag/__init__.py 0 0 100%
app/rag/agent.py 412 1 99%
app/rag/archive_upload.py 134 0 100%
app/rag/chunker.py 220 4 98%
app/rag/doc_dates.py 18 0 100%
app/rag/folder_summaries.py 134 0 100%
app/rag/git_sources.py 56 0 100%
app/rag/importer.py 671 9 99%
app/rag/llm.py 246 1 99%
app/rag/md_images.py 63 0 100%
app/rag/overview.py 71 0 100%
app/rag/pdf.py 119 0 100%
app/rag/prompts.py 154 0 100%
app/rag/retriever.py 248 3 99%
app/rag/scaffolding.py 55 0 100%
app/rag/source_removal.py 41 0 100%
app/rag/sources_meta.py 16 0 100%
app/rag/suggestions.py 27 0 100%
app/rag/summarizer.py 49 0 100%
app/schemas.py 422 0 100%
--------------------------------------------------
TOTAL 5430 22 99%
coverage gate: app/ 99% (>90%) OK
All checks passed!
0 errors, 0 warnings, 0 informations
WARNING: there is a new pyright version available (v1.1.411 -> v1.1.414).
Please install the new version or set PYRIGHT_PYTHON_FORCE_VERSION to `latest`
validation OK
.agents/reports/137_pdf_image_context/137_pdf_image_context__05_e2e_pdf_image_context.a1.err
View File
.agents/reports/137_pdf_image_context/137_pdf_image_context__05_e2e_pdf_image_context.a1.md
+21
View File
@@ -0,0 +1,21 @@
Task 05 is complete — everything is 100% green.
**Implemented**
- `tests/e2e/mock_llm.py`: new `IMAGE_DESCRIPTION_CONTEXT_ANSWER` (the phase-137 witness — names the document + page, keeps the diagram wording); `IMAGE_DESCRIPTION_MODE` branch returns it only when the text part carries the `Extracted from ` provenance header (bare path byte-identical); `_seed_context_line` also accepts the `From …` first line so the existing `CONTEXT_LINE_TRIGGER` branch echoes the task-04 display rendering; both behaviors documented in the module docstring (house rule).
- `tests/e2e/test_pdf_image_context.py` (new, isolated, port 8154): 4 tests — (1) forward path: `image_context` starts with `Extracted from e2e-pdf-imgctx/report.pdf (page 2):` + page text, `content` = context + context-aware reply, bare reply absent; (2) chat display: answer quotes `From e2e-pdf-imgctx/report.pdf (page 2)`, no "stands alone"; (4→3rd, runs pre-reseed) seed-1 KB is exactly PDF + its one image (standalone-parity witness); (3) seed 2 (markdown referencing the synthetic path): both block kinds, document first, display line carries `From … (page 2); Referenced in e2e-pdf-imgctx-md/notes.md`.
- Minimal ripple edits (embedded-image describe calls carry the provenance context by phase design): `test_pdf_support.py` seeded-fixture content assertion → context-aware reply; `test_garbage_pdf.py` embedded-image leg same (screenshot-fallback leg untouched — `context=None`, bare reply).
**Test / lint / coverage (all green)**
- `uv run pytest tests/e2e/test_pdf_image_context.py -v --no-cov` → 4 passed
- `uv run pytest tests/e2e/test_pdf_support.py tests/e2e/test_image_anchor_context.py tests/e2e/test_image_documents.py -v --no-cov` → 14 passed (anchor + documents UNEDITED)
- `uv run pytest tests/e2e/test_garbage_pdf.py -v --no-cov` → 3 passed
- `uv run pytest --cov=app --cov-report=term-missing` → 3272 passed, `app/` 99% (>90%)
- `uv run ruff check .` → all passed; `uv run pyright` → 0 errors
**Docs:** no doc impact for task 05 (test-only; the feature bullet was added to README by tasks 01–04).
**Deviations (task-file text was stale, both forced by code reality)**
- "UNEDITED" for `test_pdf_support.py` is unsatisfiable: since task 02 its embedded image's describe call carries the `Extracted from` context, so the canned reply necessarily moved to the context-aware one (one assertion + import). The load-bearing bare-path parity stays pinned by the two unedited suites.
- One mock line beyond "NO other mock change": `_seed_context_line` had a `Referenced in `-only prefix guard, so without accepting `From …` the test 2/3 display assertions could never pass.
**Next pending task:** none — task 05 is the last of phase 137; `todo/` holds only this phase.
.agents/reports/137_pdf_image_context/137_pdf_image_context__05_e2e_pdf_image_context.a1.validate
+117
View File
@@ -0,0 +1,117 @@
........................................................................ [ 2%]
........................................................................ [ 4%]
........................................................................ [ 6%]
........................................................................ [ 8%]
........................................................................ [ 11%]
........................................................................ [ 13%]
........................................................................ [ 15%]
........................................................................ [ 17%]
........................................................................ [ 19%]
........................................................................ [ 22%]
........................................................................ [ 24%]
........................................................................ [ 26%]
........................................................................ [ 28%]
........................................................................ [ 30%]
........................................................................ [ 33%]
........................................................................ [ 35%]
........................................................................ [ 37%]
........................................................................ [ 39%]
........................................................................ [ 41%]
........................................................................ [ 44%]
........................................................................ [ 46%]
........................................................................ [ 48%]
........................................................................ [ 50%]
........................................................................ [ 52%]
........................................................................ [ 55%]
........................................................................ [ 57%]
........................................................................ [ 59%]
........................................................................ [ 61%]
........................................................................ [ 63%]
........................................................................ [ 66%]
........................................................................ [ 68%]
........................................................................ [ 70%]
........................................................................ [ 72%]
........................................................................ [ 74%]
........................................................................ [ 77%]
........................................................................ [ 79%]
........................................................................ [ 81%]
........................................................................ [ 83%]
........................................................................ [ 85%]
........................................................................ [ 88%]
........................................................................ [ 90%]
........................................................................ [ 92%]
........................................................................ [ 94%]
........................................................................ [ 96%]
........................................................................ [ 99%]
................................ [100%]
=============================== warnings summary ===============================
.venv/lib/python3.13/site-packages/fastapi/testclient.py:1
/var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
from starlette.testclient import TestClient as TestClient # noqa
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
================================ tests coverage ================================
_______________ coverage: platform linux, python 3.13.13-final-0 _______________
Name Stmts Miss Cover
--------------------------------------------------
app/__init__.py 1 0 100%
app/api/__init__.py 0 0 100%
app/api/auth.py 52 0 100%
app/api/chat.py 268 1 99%
app/api/chat_images.py 50 0 100%
app/api/chats.py 110 0 100%
app/api/config.py 13 0 100%
app/api/doc_drafts.py 99 0 100%
app/api/docs.py 179 1 99%
app/api/git_sources.py 259 0 100%
app/api/health.py 10 0 100%
app/api/steering.py 42 0 100%
app/api/suggestions.py 33 0 100%
app/api/sync.py 183 0 100%
app/api/tokens.py 40 0 100%
app/api/ui_settings.py 55 0 100%
app/config.py 256 0 100%
app/core/__init__.py 0 0 100%
app/core/auth.py 45 0 100%
app/core/caching.py 124 0 100%
app/core/debugging.py 29 2 93%
app/core/docs_push.py 39 0 100%
app/core/errors.py 5 0 100%
app/core/logging.py 13 0 100%
app/core/rate_limit.py 44 0 100%
app/core/security_headers.py 20 0 100%
app/core/theming.py 38 0 100%
app/core/tokens.py 44 0 100%
app/db.py 22 0 100%
app/main.py 68 0 100%
app/models.py 133 0 100%
app/rag/__init__.py 0 0 100%
app/rag/agent.py 412 1 99%
app/rag/archive_upload.py 134 0 100%
app/rag/chunker.py 220 4 98%
app/rag/doc_dates.py 18 0 100%
app/rag/folder_summaries.py 134 0 100%
app/rag/git_sources.py 56 0 100%
app/rag/importer.py 671 9 99%
app/rag/llm.py 246 1 99%
app/rag/md_images.py 63 0 100%
app/rag/overview.py 71 0 100%
app/rag/pdf.py 119 0 100%
app/rag/prompts.py 154 0 100%
app/rag/retriever.py 248 3 99%
app/rag/scaffolding.py 55 0 100%
app/rag/source_removal.py 41 0 100%
app/rag/sources_meta.py 16 0 100%
app/rag/suggestions.py 27 0 100%
app/rag/summarizer.py 49 0 100%
app/schemas.py 422 0 100%
--------------------------------------------------
TOTAL 5430 22 99%
coverage gate: app/ 99% (>90%) OK
All checks passed!
0 errors, 0 warnings, 0 informations
WARNING: there is a new pyright version available (v1.1.411 -> v1.1.414).
Please install the new version or set PYRIGHT_PYTHON_FORCE_VERSION to `latest`
validation OK