phase: 148_pdf_pages_in_sources
**Phase 148 — final verification pass: complete, 100% green.**
- **Verified (tasks 01–04 already implemented):** page render/persist at index (`_persist_page_copies`, fail-soft, stale-replace, qualifier-loss, prune hook via `_delete_page_copies`); public `GET /api/documents/{id}/pages/{n}` (uuid4-credential, 404-all-non-servable, `image/png` + cache header); `DocContent.page_urls` additive-omitted; shared `renderDocument` strip (lazy imgs, "Page N" captions, text + Summary suppressed, onerror note, createElement-only); migration 0026; `pdf_page_dir` setting; `.env.example` entry
- **Completion criteria:** (1) pictorial + fallback classes get `p{N}.png` + `page_count`, text-heavy gets neither — pinned, pass; (2) unchanged re-sync renders zero (fast path) — pass; (3) changed re-index replaces copies, qualifier-loss/prune delete — pass; (4) route 404 matrix + payload key absent-never-null — pass; (5) modal AND /document.html strip with suppression, text-heavy byte-identical — E2E pass; (6) see below; (7) no completed-phase drift — full suite green
- **Tests:** `uv run pytest --cov=app --cov-report=term-missing` → **3461 passed, 0 failed**, `app/` coverage **99%** (>90%); `uv run pytest tests/e2e/test_pdf_pages_in_sources.py -v --no-cov` → **4 passed** in isolation; `uv run ruff check .` clean; `uv run pyright` → 0 errors, 0 warnings
- **Documentation:** README had **1 pre-existing inconsistency** — the Production Deployment section's standalone `podman run` example (and compose note) omitted the **required** `BOR_ADMIN_PASSWORD`/`BOR_SESSION_SECRET` (the fail-loud boot gate the README's own Admin section documents — the container would refuse to start); fixed with the two `-e` flags + a note for the compose path. All else verified consistent: 70/70 `.env.example` vars documented, all defaults match `config.py`, pages table matches routes, Containerfile/esbuild/uid-10001/healthcheck/entrypoint claims accurate, QA commands match pytest config. **AGENTS.md: verified consistent — no changes needed**
- **Notable:** repo uses `Containerfile` (Podman) — README's `podman build` example is valid as-is; no live infrastructure touched
- **Next pending phase:** `149_sync_progress_box`
This commit is contained in:
@@ -0,0 +1,41 @@
|
||||
# Phase 148 — pictorial PDFs show their PAGES in the sources, not their page descriptions
|
||||
|
||||
**Source:** `TODO.md` L5 — "For primarily pictographic PDFs it's not super helpful to see the summaries of the pages. I want to see the PDF pages in the sources. For example, the test_sources/ikea_instructions.tar are really only helpful if I can see the instruction pages, a description won't really help me. I still want to be able to ask questions, so the summary and embedding is helpful, but not when shown to the user."
|
||||
**Story:** n/a (TODO-derived phase, Protocol B append).
|
||||
**Context:** the owner's named fixture is the **pictorial class** — `test_sources/ikea_instructions.tar` → `besta-frame-white__AA-1272068-11-100.pdf`: 24 pages, USABLE text layer (`is_usable_pdf_text` True) but 20 of 24 pages carry <100 stripped chars (3 dense text pages, 3k–7.3k chars) — so the phase-133 screenshot fallback does NOT fire for it; the phase-140 pictorial pass (`_pictorial_pdf_pass`, gated on `pdf_pictorial` — default ON — + `pictorial_pages(texts, min_chars=100)`) describes its 20 sparse pages and appends `[Page N illustration: …]` blocks. Today the document viewer (`frontend/assets/document.js` — `renderDocument` L172, the shared core of the doc modal AND `/document.html`) shows such a PDF as its raw text content in `<pre class="doc-raw">` + the labeled Summary panel — the descriptions the owner says are "not … helpful" to a human. The rendering machinery already exists: `app/rag/pdf.py` `render_pdf_pages(pdf_path, dpi=150, pages=None)` (L311 — every page to PNG, `b""` placeholder on a per-page failure, `[]` for an unopenable file; the phase-133/140 callers). The persistent-copy convention to follow: phase 122's image copies — `settings.image_dir` (app/config.py L397, raw string, `expanduser`'d by the importer), `<doc-id>.<ext>` naming, written only after success, stale copy deleted before replacement (importer ~L2790), deleted on prune (importer `_prune` ~L2924), because "the source file is disposable: uploads are replaced on every upload, git checkouts are re-cloned, local dirs are user-edited". The viewer payload: `DocContent` (app/schemas.py — `image_url`'s additive-omission serializer is the wire precedent; the content endpoint is `GET /api/documents/content`, app/api/docs.py L165 — no doc id in the payload today, so page URLs must ride as full routes). The bytes route precedent: `GET /api/documents/{doc_id}/image` (app/api/docs.py L216 — PUBLIC, the unguessable server-minted `uuid4` IS the credential — phase-124 A4, visible on the shared page; 404 `document not found` for every non-servable case; no path parameter reaches the filesystem — the path comes from the ROW).
|
||||
**Locked (owner delegation 2026-09-30, chat — "pick the best choices"):**
|
||||
- **Eligibility** — a PDF gets rendered pages iff the sync ALREADY decided its text is not the useful view: the phase-133 fallback fired (unusable text layer — the screenshot class) OR the phase-140 pictorial pass ran and described ≥1 page (the sparse-page class — the IKEA fixture). Text-heavy PDFs (usable layer, zero candidates) keep the text view; `BOR_PDF_PICTORIAL=0` keeps its kill-switch semantics (no pictorial descriptions → no page strip for the sparse class — the toggle off means the pre-phase-148 viewer for those PDFs).
|
||||
- **Storage** — rendered at INDEX time (phase 147's fast path makes this cost one-time-per-real-change), persisted as PNGs in a NEW `settings.pdf_page_dir` (default `~/bor-sources/pages`, `BOR_PDF_PAGE_DIR` — the `image_dir` convention) named `<doc-id>-p{N}.png` (N 1-based, every page of the PDF); `documents.page_count` (migration 0026, nullable Int) = the PDF's page count (a per-page render failure leaves a gap — the route 404s that N, the viewer shows its unavailable note); stale copies deleted on re-index, pruned, and on a document that stops qualifying.
|
||||
- **Display** — the shared `renderDocument` core: for a doc with `page_urls`, the page strip (lazy `<img>`s, "Page N" captions) REPLACES the text content AND the Summary panel (the owner: the description "is helpful … but not when shown to the user" — it stays embedded + summarized for Q&A, just not displayed); the meta badge row is untouched; both surfaces (modal + full page) get it through the one core.
|
||||
|
||||
## Objective
|
||||
A pictorial-class PDF (the IKEA assembly-instruction class) opens in the sources with its actual pages — every page as a rendered image — while text-heavy PDFs, images, and text documents render byte-identically to pre-phase, and the Q&A machinery (embeddings, summaries) is untouched.
|
||||
|
||||
## Dependencies
|
||||
- `147_sync_changed_only` (todo) — REAL coupling: the page render happens in the PDF branch's new/changed path, which phase 147's fast path makes one-time-per-real-file-change (without it, an unchanged pictorial PDF would re-render its pages every sync — the same cost the owner complained about).
|
||||
|
||||
## Design (shared by all tasks — the executor reads this, not the chat)
|
||||
- **Index (task 01):** in `_index_pdf_file`, AFTER the final content is decided (fallback/pictorial passes done) and the doc is flushed (it has an id): when `new_or_changed` (reaching this code IS new/changed — the fast path returned earlier) AND eligible (the fallback fired, or the pictorial pass returned non-None): `pages = render_pdf_pages(full_path)` (every page, 150 dpi — the existing constant); delete this doc's stale page copies first (the image-copy stale-replacement convention); write each non-empty PNG to `pdf_page_dir` (created on demand) as `<doc-id>-p{N}.png`; `doc.page_count = len(doc)` (the PDF's page count — gaps tolerated). NOT eligible, or zero pages rendered (unopenable file — `render_pdf_pages` `[]`): `doc.page_count = None` + delete any stale copies (a doc that stops qualifying loses its strip). Fail-soft: a render/write exception → `page_count = None`, warning, the document indexes exactly as today. In `_prune` (and the PDF parent's embedded-image deletion site), a deleted page-qualifying doc's copies are removed (a shared `_delete_page_copies(doc)` helper — the `_delete_image_copy` convention; `source_removal.py` deletes rows only — the image-copy convention there — and is untouched).
|
||||
- **Serve (task 02):** `GET /api/documents/{doc_id}/pages/{n}` — public (the image route's trust model verbatim: the unguessable `uuid4` is the credential; a shared chat can cite the page), `n: int` path parameter ≥ 1; 404 `document not found` for: unparseable id, unknown doc, `page_count` NULL, `n` outside `1..page_count`, missing file on disk (defensive — the viewer's onerror note); the path is built from the ROW's id + `n` (no traversal surface — `n` is an int); `image/png` + `Cache-Control: private, max-age=3600` (the image route's header). `DocContent.page_urls: list[str] | None = None` — the content endpoint fills `[f"/api/documents/{doc.id}/pages/{i}" for i in 1..doc.page_count]`; the serializer drops the key when None (the `image_url` pattern — never `null`, absent for text/image docs and non-qualifying PDFs).
|
||||
- **View (task 03):** `renderDocument` (the shared core — modal + page): `doc.page_urls` a non-empty array → a `.doc-pages` strip: one `<figure class="doc-page">` per URL — `<img loading="lazy" src=url alt="Page N of {doc.title}">` + `<figcaption>Page N</figcaption>`; the markdown/`.doc-raw` content block is NOT rendered for this doc, and the Summary panel is NOT rendered (the owner's contract — the text stays in the DB for Q&A, it is simply not displayed); the meta badge row renders as today; `onerror` on an `<img>` swaps its figure for a "Page N unavailable" note (the phase-122 image-unavailable idiom, `role="status"`); `createElement`-only (the XSS contract — never innerHTML from doc data). CSS: `.doc-pages` in the existing doc content column (the ≤46rem core), figures stacked, `img { max-width: 100% }`, theme vars for border/surface/ink (both themes), `overflow-wrap` not needed (no text flows), contrast AA, `prefers-reduced-motion` irrelevant (no motion).
|
||||
|
||||
## Tasks
|
||||
1. `01_page_render_at_index.md` — `pdf_page_dir` setting + `page_count` migration/model + the PDF-branch render/persist/cleanup + the prune hook + the unit matrix.
|
||||
2. `02_page_route_and_payload.md` — the public page-bytes route + `DocContent.page_urls` + the endpoint + unit/integration tests.
|
||||
3. `03_viewer_page_strip.md` — the `renderDocument` strip (content + summary suppressed for qualifying docs) + the CSS.
|
||||
4. `04_e2e_pdf_pages.md` — the dedicated E2E: a synthetic pictorial PDF shows its pages in the modal + full page; a text-heavy PDF renders byte-identical.
|
||||
|
||||
## Testing & Quality
|
||||
- Unit: the eligibility matrix × {new, changed, unchanged-fast-path, unopenable}; copy naming/stale-cleanup/prune-deletion; the route matrix; the payload omission rule.
|
||||
- Integration: the content endpoint returns `page_urls` for a seeded page-qualifying doc and omits the key otherwise.
|
||||
- E2E: one dedicated Playwright file, run in isolation, mock-only (the house PyMuPDF fixture recipe).
|
||||
- Coverage: **>90%** on new/modified code (`app/`).
|
||||
|
||||
## Completion Criteria
|
||||
- [ ] a pictorial-class PDF (IKEA shape: usable layer, ≥1 sparse page) and a fallback-class PDF (unusable layer) get `<doc-id>-p{N}.png` copies for every page + `page_count` = the page count; a text-heavy PDF gets neither
|
||||
- [ ] an unchanged qualifying PDF's second sync (phase 147 fast path) re-renders nothing (the copies persist, zero render calls)
|
||||
- [ ] re-indexing a changed PDF replaces its copies (no stale `p{N}` orphans); a PDF that stops qualifying loses them; pruning deletes them
|
||||
- [ ] `GET /api/documents/{id}/pages/{n}` serves the PNG (public, uuid4-credential model) and 404s every non-servable case; `DocContent.page_urls` is present for qualifying docs and the key is ABSENT otherwise
|
||||
- [ ] the modal AND /document.html show the page strip for qualifying docs (lazy imgs, "Page N" captions) with the text content + Summary panel suppressed; every other document renders byte-identical to pre-phase
|
||||
- [ ] `uv run pytest --cov=app --cov-report=term-missing` green, `app/` coverage >90%; `uv run pytest tests/e2e/test_pdf_pages_in_sources.py -v --no-cov` green in isolation; `uv run ruff check . && uv run pyright` clean
|
||||
- [ ] no behavior change in completed phases (the fallback/pictorial content, `content_hash`, embeddings/summaries, the image route, the doc-edit/date/summary admin APIs)
|
||||
@@ -0,0 +1,37 @@
|
||||
# Task 01 — render qualifying PDF pages at index time: the persistent copies + `page_count`
|
||||
|
||||
**Phase:** `148_pdf_pages_in_sources` · **Source:** `TODO.md:5` — "For primarily pictographic PDFs it's not super helpful to see the summaries of the pages. I want to see the PDF pages in the sources. For example, the test_sources/ikea_instructions.tar are really only helpful if I can see the instruction pages…"
|
||||
**Story:** n/a
|
||||
|
||||
## Objective
|
||||
A qualifying PDF (fallback class or pictorial class) has every page rendered to a persistent PNG at index time — one-time per real file change (phase 147's fast path) — with `documents.page_count` recording the page count, stale copies cleaned on re-index/qualifier-loss/prune, and a fail-soft posture: a render failure never breaks the document's indexing.
|
||||
|
||||
## Work
|
||||
1. `app/config.py` — beside `image_dir` (L397): `pdf_page_dir: str = "~/bor-sources/pages"` (`BOR_PDF_PAGE_DIR`) with the house docstring: where rendered PDF page PNGs are copied for serving (phase 148) — raw string, `Path.expanduser()` applied by the importer (the `sources_dir`/`upload_dir`/`image_dir` convention); deliberately separate from `sources_dir`/`upload_dir` (git checkouts re-cloned, uploads replaced, local dirs user-edited — the served copy must outlive the source file, the `image_dir` rationale).
|
||||
2. `alembic/versions/0026_documents_page_count.py` — `revision = "0026"`, `down_revision = "0025"`: `op.add_column("documents", sa.Column("page_count", sa.Integer(), nullable=True))` (no index — a display fact, never a query surface); downgrade drops it. `app/models.py` — `Document.page_count: Mapped[int | None] = mapped_column(Integer, default=None)` + house docstring: the number of rendered page PNGs persisted in `pdf_page_dir` as `<doc-id>-p{N}.png` (phase 148) — the PDF's page count for a page-qualifying document (the phase-133 fallback class or the phase-140 pictorial class); NULL for every other document (text, images, text-heavy PDFs); set/cleared ONLY by the importer's PDF branch (+ prune cleanup deletes the copies).
|
||||
3. `app/rag/importer.py` — `_index_pdf_file` (L1181), in the NEW/CHANGED path AFTER `session.flush()` (the doc has its id) and after the final content is decided (the fallback/pictorial passes have run — the eligibility signals are local: `fallback_fired` = the phase-133 gate fired (the `summary.pdfs_screenshotted += 1` site), `pictorial_described` = `_pictorial_pdf_pass` returned non-None):
|
||||
- `eligible = fallback_fired or pictorial_described`;
|
||||
- when eligible: `pages = render_pdf_pages(full_path)` (every page — `pages=None`, 150 dpi, the existing helper's contract: `b""` placeholder at a failed page's position, `[]` for an unopenable file); FIRST delete this doc's existing page copies (the stale-replacement convention — a changed PDF's old page set, e.g. a 30-page PDF that became 24, must not leak `p25..p30`); then write each non-empty PNG to `Path(llm.settings.pdf_page_dir).expanduser() / f"{doc.id}-p{N}.png"` (N 1-based, list-index aligned — the helper's "list index i is page i+1" default contract; the directory is `mkdir(parents=True, exist_ok=True)` on demand — the `image_dir` pattern); `doc.page_count = total_pages` where `total_pages` is the PDF's page count (the `fitz` doc length — re-derive via a tiny helper or `len(pages)` when the file opened: `render_pdf_pages` returns one slot per page, so `len(pages)` IS the page count for an openable file; `[]` → the unopenable case);
|
||||
- when NOT eligible, or `pages == []` (unopenable), or ALL pages failed (zero non-empty PNGs): `doc.page_count = None` + delete any existing copies for this doc (a document that STOPPED qualifying loses its strip — e.g. a new version gained a real text layer);
|
||||
- **fail-soft:** wrap the render/persist block — any exception → `page_count = None` + `logger.warning` (the house wording style) + the document proceeds to chunk/embed/summary EXACTLY as pre-phase (the render is a display concern — it never gates indexing); a per-page `b""` is NOT an exception (the gap is served as a 404 later, the viewer's note covers it).
|
||||
- the UNCHANGED path (phase 147's fast path — `_unchanged_doc`) does NOT touch pages or `page_count` (the copies persist; zero render work — assert this).
|
||||
4. `app/rag/importer.py` — a shared `_delete_page_copies(doc, page_dir) -> None` helper (the `_delete_image_copy` convention, L2626: missing files are a silent no-op, never an error): delete `<doc-id>-p1..p{page_count}.png` (iterate 1..`doc.page_count` — the row knows its own extent; a NULL/zero count is a no-op). Call sites: (a) the stale-replacement in step 3 (pass the doc BEFORE its `page_count` changes), (b) `_prune` (L2870) — beside the existing `_delete_image_copy(doc.image_path)` call (~L2924) for a pruned page-qualifying doc, (c) the embedded-image parent-PDF prune path (the pruned parent's copies too — the same site that deletes the parent's embedded-image copies). `app/rag/source_removal.py` is UNTOUCHED (row-only deletion — the image-copy convention there).
|
||||
5. `tests/unit/` — the importer unit home:
|
||||
- **eligibility × lifecycle matrix** (fake llm, tmp `pdf_page_dir`, the house PyMuPDF fixture recipes — the `test_pictorial_pdf.py` builder for the pictorial class, a textless/blank-page PDF for the fallback class, a dense-text PDF for the negative): new index of a qualifying PDF → copies `p1..p{N}` exist (N = page count), `page_count == N`, the PNG bytes are non-empty; re-run unchanged (phase 147 fast path) → ZERO render calls (count via monkeypatched `render_pdf_pages` or call counter), copies + `page_count` untouched; change the file (append a page to the fixture) → re-index → the OLD copy set replaced (a shrunk PDF leaves no stale `p{N}`), `page_count` updated;
|
||||
- **qualifier loss:** a qualifying PDF's new version has a dense text layer (zero candidates, `pdf_pictorial` on) → `page_count` NULL + copies deleted;
|
||||
- **unopenable:** `render_pdf_pages` returns `[]` (monkeypatched) → `page_count` NULL, no copies, no exception, the doc still indexed (chunks/summary ran);
|
||||
- **per-page gap:** one page's render yields `b""` (monkeypatched) → the other copies written, `page_count` = the full count (the gap is a display-level 404, pinned in task 02);
|
||||
- **render exception:** the writer raises → `page_count` NULL, warning logged (caplog), the doc indexed fully (fail-soft);
|
||||
- **prune:** a qualifying doc pruned (its file removed from the source) → row gone + copies gone (no orphans in the tmp dir);
|
||||
- **settings:** `pdf_page_dir` default `~/bor-sources/pages`, env override `BOR_PDF_PAGE_DIR` (the settings test home).
|
||||
|
||||
## Testing & Quality
|
||||
- Unit: the matrix above; the phase-147 fast-path pins pass UNCHANGED (the fast path skips the render block by construction — it returns before the branch); the pre-phase PDF-branch pins (fallback content, pictorial blocks, `content_hash` over the final content, the embedded-image pass) pass UNCHANGED.
|
||||
- Coverage: **>90%** on this task's new/modified code (`app/`).
|
||||
|
||||
## Completion Criteria
|
||||
- [ ] qualifying PDFs (both classes) get `<doc-id>-p{N}.png` for every renderable page + `page_count` = the page count; non-qualifying PDFs, text docs, and images get neither
|
||||
- [ ] unchanged re-syncs render nothing; changed re-syncs replace the copy set without orphans; qualifier-loss and prune delete the copies
|
||||
- [ ] a render failure is fail-soft: the document indexes exactly as pre-phase, `page_count` NULL, a warning is the signal
|
||||
- [ ] `settings.pdf_page_dir` follows the `image_dir` convention (raw string, expanduser'd at use); migration 0026 round-trips
|
||||
- [ ] `uv run pytest` green; `uv run pytest --cov=app --cov-report=term-missing` — `app/` coverage >90%; `uv run ruff check . && uv run pyright` clean
|
||||
@@ -0,0 +1,29 @@
|
||||
# Task 02 — serve the pages: the public route + `DocContent.page_urls`
|
||||
|
||||
**Phase:** `148_pdf_pages_in_sources` · **Source:** `TODO.md:5` — "I want to see the PDF pages in the sources. For example, the test_sources/ikea_instructions.tar are really only helpful if I can see the instruction pages, a description won't really help me."
|
||||
**Story:** n/a
|
||||
|
||||
## Objective
|
||||
The viewer payload names every rendered page (`page_urls`, full routes, additive-omitted), and a public bytes route serves one page PNG at a time — the image route's trust model and 404 posture verbatim.
|
||||
|
||||
## Work
|
||||
1. `app/api/docs.py` — new route beside `get_document_image` (L216): `@router.get("/documents/{doc_id}/pages/{n}", response_class=FileResponse)` → `def get_document_page(doc_id: str, n: int, db = Depends(get_db)) -> FileResponse` (the `{n}` int path parameter is parsed by FastAPI — a non-int 422s at the router, mirroring how the image route's unparseable `doc_id` maps to its 404 below):
|
||||
- the docstring: PUBLIC (the phase-124 A4 trust model verbatim — a shared chat can cite a page figure, so the bytes serve ANONYMOUSLY; the document id is an unguessable server-minted `uuid4` — IT IS the credential, no enumeration value — exactly like the image route and the saved-chat id; the document's TEXT stays user-gated: only the page bytes are public);
|
||||
- 404 `detail="document not found"` (the router's unknown-document shape) for EVERY non-servable case: unparseable `doc_id` (the `uuid.UUID` try/except → 404, not 422 — the image route's mapping), an unknown doc, `page_count` NULL (a non-qualifying doc — the route knows nothing of its pages), `n < 1` or `n > doc.page_count` (out of range — the int parameter makes `n < 0`/fractional impossible; a gap — a page whose render failed — hits the missing-file 404 below), or the file on disk missing (defensive — the copy was lost; the viewer's onerror note covers it);
|
||||
- the path is built ONLY from the row's id + `n` (`Path(settings.pdf_page_dir).expanduser() / f"{doc.id}-p{n}.png"` — `settings` via the house import) — no request value reaches a filesystem path (no traversal surface, the image route's property);
|
||||
- `FileResponse(page_file, media_type="image/png", headers={"Cache-Control": "private, max-age=3600"})` (the PNG is always PNG — `render_pdf_pages`'s `tobytes("png")`; the image route's header, the content-hashed-bytes rationale).
|
||||
2. `app/schemas.py` — `DocContent`: add `page_urls: list[str] | None = None` + docstring (phase 148 — the rendered-page bytes routes `["/api/documents/<id>/pages/<i>" for i in 1..page_count]` for a page-qualifying PDF; ABSENT from the wire for every other document — `None` → dropped by the serializer, never `null`, the `image_url` omission precedent); extend `_serialize` to pop `page_urls` when None (beside the existing `image_url` pop).
|
||||
3. `app/api/docs.py` — `get_document_content` (L165, the `GET /documents/content` endpoint): when `doc.page_count` is non-NULL, fill `page_urls = [f"/api/documents/{doc.id}/pages/{i}" for i in range(1, doc.page_count + 1)]` (the `/api` prefix — the `image_url` field carries the same full-route shape; check `image_url`'s exact construction and mirror it byte-for-byte in convention).
|
||||
4. `tests/unit/` + the docs-endpoint integration home:
|
||||
- **route matrix** (the TestClient idiom the image-route tests use): a seeded qualifying doc (row with `page_count=2` + real PNG files in a tmp `pdf_page_dir`) → `GET .../pages/1` and `/2` → 200, `content-type: image/png`, the `Cache-Control` header, the body bytes == the stored file's bytes; `n=0`, `n=3` → 404 `document not found`; a text doc row / an image doc row / a PDF row with `page_count=NULL` → 404 on any `n`; a qualifying row whose file was deleted on disk → 404; a non-uuid `doc_id` (`"abc"`) → 404 (not 422);
|
||||
- **payload:** the content endpoint for a qualifying doc → `page_urls` == the exact route list (1-based, count == `page_count`); a text doc / an image doc / a non-qualifying PDF → the `page_urls` KEY is ABSENT from the JSON (assert `"page_urls" not in data` — the omission pin, the `image_url` pattern);
|
||||
- **no traversal:** the route is structurally impossible to point at an arbitrary path (the int parameter + row-derived path — assert the code builds the path from `doc.id` + `n` only, pinned by the out-of-range 404s).
|
||||
|
||||
## Testing & Quality
|
||||
- Unit + integration: the matrices above; every pre-existing `DocContent` pin passes UNCHANGED (a non-qualifying doc's payload is byte-identical — the key is absent, not null); the image route's tests are untouched.
|
||||
- Coverage: **>90%** on this task's new/modified code (`app/`).
|
||||
|
||||
## Completion Criteria
|
||||
- [ ] `GET /api/documents/{id}/pages/{n}` serves the stored PNG (public, uuid4-credential, `image/png`, the image route's cache header) and 404s EVERY non-servable case with `document not found`
|
||||
- [ ] `DocContent.page_urls` is the exact 1-based route list for qualifying docs and the key is ABSENT (never `null`) for every other document
|
||||
- [ ] `uv run pytest` green; `uv run pytest --cov=app --cov-report=term-missing` — `app/` coverage >90%; `uv run ruff check . && uv run pyright` clean
|
||||
@@ -0,0 +1,32 @@
|
||||
# Task 03 — the viewer shows the pages: the strip replaces the text and the summary panel
|
||||
|
||||
**Phase:** `148_pdf_pages_in_sources` · **Source:** `TODO.md:5` — "For primarily pictographic PDFs it's not super helpful to see the summaries of the pages. I want to see the PDF pages in the sources. … I still want to be able to ask questions, so the summary and embedding is helpful, but not when shown to the user."
|
||||
**Story:** n/a
|
||||
|
||||
## Objective
|
||||
For a document whose payload carries `page_urls`, the shared `renderDocument` core renders the page strip — every page as a lazy image with a "Page N" caption — and suppresses the text content AND the Summary panel (the descriptions stay in the DB for Q&A, just not displayed). Every other document renders byte-identically to pre-phase, on both surfaces (the doc modal and `/document.html`).
|
||||
|
||||
## Work
|
||||
1. `frontend/assets/document.js` — `renderDocument` (L172 — the ONE shared core; the modal and the full page both call it, no per-surface copy):
|
||||
- the gate: `const pageUrls = Array.isArray(doc.page_urls) ? doc.page_urls : null;` — a non-empty array triggers the strip; `null`/absent/empty → the EXACT pre-phase rendering (assertable byte-identity for every other doc);
|
||||
- the strip: a `<div class="doc-pages">` container (replacing what would become the content slot) holding one `<figure class="doc-page">` per URL — `<img loading="lazy" src={url}>` with `alt = `Page ${i} of ${doc.title}`` (the WCAG alt contract — the phase-122 image idiom, the title as the fallback context) and a `<figcaption class="doc-page-num">Page ${i}</figcaption>` (i 1-based over `page_urls`);
|
||||
- **suppression:** when the strip renders, the markdown/`.doc-raw` content block is NOT built AND the labeled Summary panel (`.doc-summary`, phase 36 — including its admin edit affordance wiring, which is moot without the panel) is NOT built — the owner's contract: the page descriptions are "not … shown to the user"; the meta badge row (Created/Indexed, the phase-106 badges) renders exactly as today;
|
||||
- **unavailable page:** each `<img>` gets an `onerror` handler (the phase-122 "Image unavailable" idiom) that swaps ITS figure's `<img>` for a small "Page N unavailable" note element (`role="status"`, muted styling) — the rest of the strip is untouched (one lost copy never hides the other pages; the defensive 404s of task 02's route are the trigger);
|
||||
- **XSS contract:** `createElement`/`textContent` only — every document-derived string (title, URLs, the note) lands via properties/`textContent`, never innerHTML (the file's pinned contract — the module docstring's "XSS-safe by construction" line gains the phase-148 note).
|
||||
2. `frontend/assets/styles.css` — the strip, in the existing doc content column (the core renders into the same `#doc-content` slot the `.doc-raw`/markdown path uses — no new column):
|
||||
- `.doc-pages { display: grid; gap: <house spacing>; }` (vertical stack — one page per row, the natural reading order for instruction manuals);
|
||||
- `.doc-page { margin: 0; } .doc-page img { display: block; width: 100%; max-width: 100%; height: auto; border: 1px solid <theme border var>; border-radius: <house radius>; background: <theme surface var>; }` (a page PNG on a dark theme must sit on a non-clashing surface — the image route's figures already have a dark/light precedent to mirror);
|
||||
- `.doc-page-num { font: <house small meta style>; color: <muted ink var>; text-align: center; padding: <small>; }` (contrast ≥4.5:1 in BOTH themes — the WCAG gate);
|
||||
- the "Page N unavailable" note reuses the phase-122 image-unavailable note styling (find the existing class — the same muted note, no new pattern).
|
||||
3. Module docstring: the phase-148 paragraph (the gate, the suppression, the strip, the onerror note, both surfaces) in the house style of the file's leading changelog.
|
||||
|
||||
## Testing & Quality
|
||||
- The house tests `document.js` through E2E + the frontend emoji guard (no JS unit harness) — the rendering pins live in task 04's E2E; this task's gate is that the pre-phase suites (the doc-modal, document-page, image-document, summary-panel E2Es) pass UNCHANGED (a non-qualifying doc is byte-identical — the load-bearing property).
|
||||
- Coverage: n/a (frontend task — `app/` coverage stays >90% from tasks 01–02).
|
||||
|
||||
## Completion Criteria
|
||||
- [ ] a payload with `page_urls` renders the strip (one figure per URL, lazy imgs, "Page N" captions, alt text) with NO text content block and NO Summary panel; the meta badges are untouched
|
||||
- [ ] a payload without `page_urls` (text docs, images, text-heavy PDFs, pre-phase saved content) renders byte-identically to pre-phase on BOTH surfaces
|
||||
- [ ] one failed page image swaps only its own figure for the "Page N unavailable" note
|
||||
- [ ] no innerHTML from document data; the emoji guard stays green
|
||||
- [ ] `uv run pytest` green (all pre-existing E2Es); `uv run ruff check . && uv run pyright` clean
|
||||
@@ -0,0 +1,29 @@
|
||||
# Task 04 — the dedicated E2E: a pictorial PDF opens as its pages, a text-heavy PDF as before
|
||||
|
||||
**Phase:** `148_pdf_pages_in_sources` · **Source:** `TODO.md:5` — "I want to see the PDF pages in the sources. For example, the test_sources/ikea_instructions.tar are really only helpful if I can see the instruction pages, a description won't really help me."
|
||||
**Story:** n/a
|
||||
|
||||
## Objective
|
||||
End-to-end proof: a synthetic IKEA-shaped PDF (usable text layer, sparse pages → the pictorial class, the owner's named fixture's exact class) syncs, gets its pages, and opens in the doc modal + full page as a strip of real page images — while a text-heavy PDF renders byte-identically to pre-phase.
|
||||
|
||||
## Work
|
||||
1. `tests/e2e/test_pdf_pages_in_sources.py` — the dedicated suite (one Playwright file, run in isolation, mock-only; the module-app pattern of the phase-126/133/140 PDF suites — its own `BOR_SOURCES_DIR` tmp dir + its own port, the LLM the mock; `pdf_pictorial` ON by default — the IKEA class):
|
||||
- **Fixture builder** (the house PyMuPDF recipe — the `test_pictorial_pdf.py` builder, L214–236, adapted):
|
||||
- `pictorial.pdf` — the IKEA shape: page 1 dense text (a paragraph, >100 stripped chars — the `is_usable_pdf_text` gate passes on the whole layer), pages 2–3 SPARSE (a drawn rectangle + a line, <100 chars — the `pictorial_pages(texts, min_chars=100)` candidates — the mock vision describes them, the `PICTORIAL_PAGE_ANSWER` canned description);
|
||||
- `textheavy.pdf` — 2 dense text pages (zero candidates — the negative class).
|
||||
- Both land in a local-directory source (a `BOR_SOURCES_DIR` subdirectory — local dirs are the owner's named case; a sync run indexes them — the house sync trigger, the admin `Sync sources` click or the API the phase-126 suites use).
|
||||
- **Test 1 — `test_pictorial_pdf_shows_pages_in_modal`:** after the sync settles, assert via the DB (`SessionLocal`, the house idiom) that the pictorial doc row has `page_count == 3` and three `<id>-p{1..3}.png` files exist under the configured `pdf_page_dir`; open the doc modal from the Sources tree (the house `openDocumentModal`/tree-click idiom the doc-modal E2Es use) → inside the modal content: `.doc-pages` present with EXACTLY three `.doc-page` figures; each `<img>` is LOADED (`img.complete && naturalWidth > 0` — the route served a real PNG, not a 404); the figcaptions read `Page 1..3`; the alt texts carry the doc title; NO `.doc-raw` and NO `.doc-summary` in the modal content (the suppression contract — the mock's page descriptions are in the DB but not displayed); the meta badge row (Created/Indexed) is present.
|
||||
- **Test 2 — `test_textheavy_pdf_unchanged`:** open `textheavy.pdf`'s modal → the payload has NO `page_urls` (assert through the page: the content endpoint's JSON via the house fetch idiom, or simply the rendering) → the content renders in `.doc-raw` (the pre-phase shape — the pre-existing doc-viewer pins), the Summary panel renders (the mock summary is present) — byte-identical to pre-phase (the phase-10/26/36 pins hold).
|
||||
- **Test 3 — `test_full_page_surface`:** navigate to the doc's full page (`/document.html?source=…&path=…`, the house URL idiom — admin-logged-in, the phase-79 gate satisfied) → the SAME strip renders (one `renderDocument` core — no per-surface drift) with the same suppression.
|
||||
- **Test 4 — `test_page_route_serves_png`:** a direct `page.request.get` on the first `page_urls` entry (or a fresh page context) → 200, `content-type: image/png`, the `Cache-Control` header, a non-empty body (the bytes route's contract from the UI's perspective).
|
||||
2. Teardown: the module app stops; the tmp dirs (sources + `pdf_page_dir`) are disposable (tmp_path_factory); the suite is hermetic (own port, own dirs) so `--no-cov` isolation holds.
|
||||
|
||||
## Testing & Quality
|
||||
- E2E: the four tests above, mock-only, real rendering (PyMuPDF renders the synthetic PDF's pages for real — `naturalWidth > 0` proves the full pipeline: sync → render → persist → route → viewer).
|
||||
- Coverage: n/a (E2E task — `app/` coverage stays >90% from tasks 01–03).
|
||||
|
||||
## Completion Criteria
|
||||
- [ ] the IKEA-shaped pictorial PDF shows all 3 pages as loaded images in BOTH the modal and the full page, with captions/alt text, no text content, no summary panel
|
||||
- [ ] the text-heavy PDF renders byte-identically to pre-phase (`.doc-raw` + summary panel, no strip)
|
||||
- [ ] the page route serves a real PNG (200, `image/png`, the cache header, non-empty)
|
||||
- [ ] `uv run pytest tests/e2e/test_pdf_pages_in_sources.py -v --no-cov` green in isolation; `uv run pytest` still green (the pre-existing PDF/doc-viewer suites pass unchanged)
|
||||
Reference in New Issue
Block a user