phase: 148_pdf_pages_in_sources
**Phase 148 — final verification pass: complete, 100% green.**
- **Verified (tasks 01–04 already implemented):** page render/persist at index (`_persist_page_copies`, fail-soft, stale-replace, qualifier-loss, prune hook via `_delete_page_copies`); public `GET /api/documents/{id}/pages/{n}` (uuid4-credential, 404-all-non-servable, `image/png` + cache header); `DocContent.page_urls` additive-omitted; shared `renderDocument` strip (lazy imgs, "Page N" captions, text + Summary suppressed, onerror note, createElement-only); migration 0026; `pdf_page_dir` setting; `.env.example` entry
- **Completion criteria:** (1) pictorial + fallback classes get `p{N}.png` + `page_count`, text-heavy gets neither — pinned, pass; (2) unchanged re-sync renders zero (fast path) — pass; (3) changed re-index replaces copies, qualifier-loss/prune delete — pass; (4) route 404 matrix + payload key absent-never-null — pass; (5) modal AND /document.html strip with suppression, text-heavy byte-identical — E2E pass; (6) see below; (7) no completed-phase drift — full suite green
- **Tests:** `uv run pytest --cov=app --cov-report=term-missing` → **3461 passed, 0 failed**, `app/` coverage **99%** (>90%); `uv run pytest tests/e2e/test_pdf_pages_in_sources.py -v --no-cov` → **4 passed** in isolation; `uv run ruff check .` clean; `uv run pyright` → 0 errors, 0 warnings
- **Documentation:** README had **1 pre-existing inconsistency** — the Production Deployment section's standalone `podman run` example (and compose note) omitted the **required** `BOR_ADMIN_PASSWORD`/`BOR_SESSION_SECRET` (the fail-loud boot gate the README's own Admin section documents — the container would refuse to start); fixed with the two `-e` flags + a note for the compose path. All else verified consistent: 70/70 `.env.example` vars documented, all defaults match `config.py`, pages table matches routes, Containerfile/esbuild/uid-10001/healthcheck/entrypoint claims accurate, QA commands match pytest config. **AGENTS.md: verified consistent — no changes needed**
- **Notable:** repo uses `Containerfile` (Podman) — README's `podman build` example is valid as-is; no live infrastructure touched
- **Next pending phase:** `149_sync_progress_box`
This commit is contained in:
@@ -40,6 +40,20 @@ traversal/UUID-guessing has no filesystem surface to hit). The
|
||||
document's TEXT (the content endpoint) stays user-gated — only the
|
||||
bytes are public.
|
||||
|
||||
GET /api/documents/{doc_id}/pages/{n} — the phase-148 (task 02) PDF
|
||||
page BYTES route: ONE rendered page PNG (``<doc-id>-p{N}.png`` in the
|
||||
importer's ``pdf_page_dir`` copy — task 01) of a page-QUALIFYING
|
||||
document (the phase-133 fallback class or the phase-140 pictorial
|
||||
class). PUBLIC — the image route's phase-124 A4 trust model verbatim
|
||||
(a shared chat can cite a page figure, so the bytes serve
|
||||
ANONYMOUSLY; the unguessable ``uuid4`` document id IS the credential —
|
||||
the document's TEXT stays user-gated). ``image/png`` +
|
||||
``Cache-Control: private, max-age=3600`` (the image route's header);
|
||||
404 ``document not found`` for EVERY non-servable case (an
|
||||
unparseable id, an unknown doc, ``page_count`` NULL, ``n`` outside
|
||||
``1..page_count``, or a copy lost on disk). The path is built from the
|
||||
ROW's id + ``n`` only — no request value reaches a filesystem path.
|
||||
|
||||
PATCH /api/folders/summary — the admin folder-description editor
|
||||
(phase 97, task 03): update / create / clear a stored
|
||||
``folder_summaries`` row, marking every non-empty save
|
||||
@@ -228,6 +242,18 @@ def get_document_content(
|
||||
),
|
||||
origin_url=origin[0] if origin else None,
|
||||
origin_branch=origin[1] if origin else None,
|
||||
# Phase 148 (task 02): the rendered-page BYTES routes for a
|
||||
# page-qualifying PDF — the exact 1-based list over
|
||||
# ``1..page_count`` (the viewer's strip source, task 03; a
|
||||
# per-page render failure leaves a GAP the route 404s). ABSENT
|
||||
# (never null — the ``image_url`` omission rule) for every
|
||||
# other document, so its payload is byte-identical to
|
||||
# pre-phase.
|
||||
page_urls=(
|
||||
[f"/api/documents/{doc.id}/pages/{i}" for i in range(1, doc.page_count + 1)]
|
||||
if doc.page_count is not None
|
||||
else None
|
||||
),
|
||||
)
|
||||
|
||||
|
||||
@@ -290,6 +316,69 @@ def get_document_image(
|
||||
)
|
||||
|
||||
|
||||
@router.get("/documents/{doc_id}/pages/{n}", response_class=FileResponse)
|
||||
def get_document_page(
|
||||
doc_id: str,
|
||||
n: int,
|
||||
db: Session = Depends(get_db), # noqa: B008
|
||||
) -> FileResponse:
|
||||
"""The phase-148 (task 02) PDF page BYTES route — ONE rendered page
|
||||
PNG (``<doc-id>-p{N}.png`` in the importer's ``pdf_page_dir`` copy,
|
||||
task 01) of a page-QUALIFYING document.
|
||||
|
||||
PUBLIC (the phase-124 A4 trust model verbatim — the image route's
|
||||
precedent): a shared chat can cite a page FIGURE (the viewer's
|
||||
strip, task 03), so the bytes serve ANONYMOUSLY. The document id
|
||||
is an unguessable server-minted ``uuid4`` — IT IS the credential,
|
||||
with no enumeration value — exactly like the image route and the
|
||||
saved-chat id (phase 55). The document's TEXT stays user-gated (the
|
||||
content endpoint): only the page bytes are public.
|
||||
|
||||
404 ``document not found`` (the router's unknown-document shape) in
|
||||
EVERY non-servable case — an unparseable ``doc_id`` (the
|
||||
``uuid.UUID`` try/except maps here, not to a 422 — the image
|
||||
route's mapping: a guessed id is an unknown document), an unknown
|
||||
doc, a doc whose ``page_count`` is NULL (a non-qualifying doc — the
|
||||
route knows nothing of its pages), ``n`` outside ``1..page_count``
|
||||
(out of range — a GAP, a page whose render failed, hits the
|
||||
missing-file arm below), or a file on disk that is missing
|
||||
(defensive — the row exists, the bytes don't; the viewer's onerror
|
||||
note covers it).
|
||||
|
||||
``n`` is an ``int`` path parameter (FastAPI parses it — a non-int
|
||||
422s at the router), and the path is built ONLY from the row's id
|
||||
+ ``n`` (``pdf_page_dir/<doc.id>-p{n}.png``) — no request value
|
||||
reaches a filesystem path (no traversal surface, the image route's
|
||||
property).
|
||||
|
||||
Servable pages stream the exact stored bytes as ``image/png``
|
||||
(``render_pdf_pages``'s ``tobytes("png")`` — a page is always a
|
||||
PNG) with ``Cache-Control: private, max-age=3600`` (the image
|
||||
route's header — the bytes are content-addressed by the document
|
||||
id + page number, bustable by re-index).
|
||||
"""
|
||||
try:
|
||||
uid = uuid.UUID(doc_id)
|
||||
except ValueError:
|
||||
raise HTTPException(status_code=404, detail="document not found") from None
|
||||
doc = db.scalar(select(Document).where(Document.id == uid))
|
||||
if doc is None or doc.page_count is None:
|
||||
raise HTTPException(status_code=404, detail="document not found")
|
||||
if not 1 <= n <= doc.page_count:
|
||||
raise HTTPException(status_code=404, detail="document not found")
|
||||
page_file = Path(get_settings().pdf_page_dir).expanduser() / f"{doc.id}-p{n}.png"
|
||||
if not page_file.is_file():
|
||||
# Defensive: the row qualifies but the copy was lost (the owner
|
||||
# cleaned the page dir, the disk was wiped) — the viewer's
|
||||
# onerror note renders "Page N unavailable".
|
||||
raise HTTPException(status_code=404, detail="document not found")
|
||||
return FileResponse(
|
||||
page_file,
|
||||
media_type="image/png",
|
||||
headers={"Cache-Control": "private, max-age=3600"},
|
||||
)
|
||||
|
||||
|
||||
@router.patch("/documents/summary", response_model=SummaryResult)
|
||||
async def update_document_summary(
|
||||
payload: SummaryUpdate,
|
||||
|
||||
@@ -395,6 +395,24 @@ class Settings(BaseSettings):
|
||||
#: checkouts, re-cloned) and ``upload_dir`` (replaced on every
|
||||
#: upload): the served copy must outlive the source file.
|
||||
image_dir: str = "~/bor-sources/images"
|
||||
#: Where rendered PDF PAGE PNGs are copied for serving (phase 148,
|
||||
#: ``BOR_PDF_PAGE_DIR`` — the ``image_dir`` convention): a
|
||||
#: page-qualifying PDF (the phase-133 fallback class — unusable
|
||||
#: text layer — or the phase-140 pictorial class — the sparse-page
|
||||
#: pass described ≥1 page) gets EVERY page rendered at index time
|
||||
#: to ``<doc-id>-p{N}.png`` here (N 1-based; ``documents.page_count``
|
||||
#: is the page count — a per-page render failure leaves a gap),
|
||||
#: so the document viewer shows the ACTUAL pages instead of the
|
||||
#: page descriptions (TODO L5 — the IKEA assembly-instruction
|
||||
#: class: the descriptions help the embeddings, "but not when
|
||||
#: shown to the user"). Raw string — ``Path.expanduser()`` is
|
||||
#: applied by the importer, not here (the ``sources_dir``/
|
||||
#: ``upload_dir``/``image_dir`` convention). Deliberately separate
|
||||
#: from ``sources_dir``/``upload_dir``: git checkouts are
|
||||
#: re-cloned, uploads are replaced, local dirs are user-edited —
|
||||
#: the served copy must outlive the source file (the ``image_dir``
|
||||
#: rationale).
|
||||
pdf_page_dir: str = "~/bor-sources/pages"
|
||||
|
||||
# --- Pictorial PDF pages (phase 140: sparse pages rendered + vision-described) ---
|
||||
#: Master switch for the pictorial PDF pass (phase 140, LOCKED A3,
|
||||
|
||||
@@ -183,6 +183,21 @@ class Document(Base):
|
||||
#: embedding model embeds this alongside the vision description when
|
||||
#: the image is enriched (task 03).
|
||||
image_context: Mapped[str | None] = mapped_column(Text, default=None)
|
||||
#: The number of rendered page PNGs persisted in
|
||||
#: ``settings.pdf_page_dir`` (phase 148, task 01) as
|
||||
#: ``<doc-id>-p{N}.png`` (N 1-based): the PDF's page count for one
|
||||
#: page-QUALIFYING document — the phase-133 fallback class (unusable
|
||||
#: text layer — the screenshot class) or the phase-140 pictorial
|
||||
#: class (the sparse-page pass described ≥1 page — the IKEA
|
||||
#: assembly-instruction class). NULL for every other document
|
||||
#: (text documents, images, text-heavy PDFs) and for a qualifying
|
||||
#: PDF whose render/write failed (fail-soft — the render is a
|
||||
#: display concern, it never gates indexing). Set/cleared ONLY by
|
||||
#: the importer's PDF branch (a per-page render failure leaves a
|
||||
#: GAP — the file is absent, the page route 404s that N); the prune
|
||||
#: cleanup deletes the copies with the row. No index: a display
|
||||
#: fact, never a query surface (the ``file_hash`` convention, 0025).
|
||||
page_count: Mapped[int | None] = mapped_column(Integer, default=None)
|
||||
|
||||
chunks: Mapped[list[Chunk]] = relationship(
|
||||
back_populates="document", cascade="all, delete-orphan"
|
||||
|
||||
+201
-1
@@ -177,6 +177,29 @@ counters ``pdfs_pictorial`` / ``pdf_pictorial_pages`` are log-only
|
||||
(the sync ``detail`` API shape is untouched, the stats log-only
|
||||
convention).
|
||||
|
||||
Persistent page copies (phase 148, task 01): in the PDF branch's
|
||||
NEW/CHANGED path — after the final content is decided (the
|
||||
fallback/pictorial passes have run) and after the row's flush (the
|
||||
doc has its id) — a page-QUALIFYING PDF (the phase-133 fallback
|
||||
fired, or the phase-140 pictorial pass described ≥1 page — the
|
||||
sync's own decisions: text-heavy PDFs stay ineligible, and the
|
||||
``pdf_pictorial`` kill switch keeps its semantics) gets EVERY page
|
||||
rendered at index time (:func:`app.rag.pdf.render_pdf_pages` — 150
|
||||
dpi, every page) and persisted to ``settings.pdf_page_dir``
|
||||
(``BOR_PDF_PAGE_DIR``, the ``image_dir`` convention) as
|
||||
``<doc-id>-p{N}.png`` (N 1-based), with ``documents.page_count``
|
||||
(migration 0026) recording the page count — a per-page render
|
||||
failure leaves a GAP (no file — the route 404s that N, the viewer's
|
||||
note covers it). The existing copies are deleted before replacement
|
||||
(a shrunk PDF's old ``p{N}`` tail cannot leak), a document that
|
||||
STOPS qualifying is cleared (``page_count`` NULL + copies deleted),
|
||||
and the prune deletes the copies with the row
|
||||
(:func:`_delete_page_copies`). FAIL-SOFT: a render or write failure
|
||||
leaves ``page_count`` NULL and logs a warning — the document indexes
|
||||
EXACTLY as pre-phase (the render is a display concern, it never
|
||||
gates the indexing). The UNCHANGED fast path renders nothing (the
|
||||
copies persist — zero render work, zero orphans).
|
||||
|
||||
Image-anchor context (phase 125, task 03, LOCKED A1/A2/A3): BEFORE the
|
||||
description, the same source's importable markdown files are scanned for
|
||||
```` references that name the image (the task-02
|
||||
@@ -851,6 +874,7 @@ async def import_sources(
|
||||
summary.pruned = _prune(
|
||||
session, source_names, seen, images=llm.settings.images,
|
||||
summary=summary,
|
||||
page_dir=Path(llm.settings.pdf_page_dir).expanduser(),
|
||||
)
|
||||
summary.embed_batches = llm.embed_batches
|
||||
summary.log()
|
||||
@@ -1413,7 +1437,19 @@ async def _index_pdf_file(
|
||||
(:func:`_unchanged_doc`) — no re-extract, no re-embed, no image
|
||||
pass, the backfill/manual-date/date-refresh semantics in one
|
||||
code site — and the helper stamps the new file hash (the next
|
||||
sync takes the fast path).
|
||||
sync takes the fast path). This path renders NO pages and
|
||||
touches NO ``page_count`` (the copies persist — zero render
|
||||
work).
|
||||
* Phase 148 (task 01): in this NEW/CHANGED path, RIGHT AFTER the
|
||||
row's flush and BEFORE the chunk pipeline, a page-QUALIFYING
|
||||
PDF (the phase-133 fallback fired, or the phase-140 pictorial
|
||||
pass described ≥1 page — the signals recorded above) gets its
|
||||
every page rendered to ``settings.pdf_page_dir`` as
|
||||
``<doc-id>-p{N}.png`` and ``documents.page_count`` set
|
||||
(:func:`_persist_pdf_pages` — stale copies deleted first, a
|
||||
stopped-qualifier cleared, a per-page ``b""`` a gap). FAIL-SOFT
|
||||
— a render/write exception leaves ``page_count`` NULL + a
|
||||
warning; the document indexes exactly as pre-phase.
|
||||
* THEN the embedded images: :func:`_index_pdf_embedded_images`
|
||||
(A3 — the phase-122 pipeline, gated by the ``images`` toggle).
|
||||
|
||||
@@ -1435,12 +1471,25 @@ async def _index_pdf_file(
|
||||
# Extract the text (fail-soft: an unopenable PDF yields "" — the
|
||||
# document may still carry embedded images worth indexing).
|
||||
content = extract_pdf_text(full_path)
|
||||
# Phase 148 (task 01): the page-strip ELIGIBILITY signals — the
|
||||
# LOCKED rule, recorded where the sync makes the decision: a PDF
|
||||
# gets rendered pages iff the sync ALREADY decided its text is not
|
||||
# the useful view — the phase-133 fallback fired (unusable text
|
||||
# layer — the screenshot class) OR the phase-140 pictorial pass
|
||||
# described ≥1 page (the sparse-page class — the IKEA fixture).
|
||||
# Text-heavy PDFs (usable layer, zero candidates) stay ineligible
|
||||
# (the text view), and ``pdf_pictorial=0`` keeps its kill-switch
|
||||
# semantics (no pictorial pass → the sparse class stays ineligible
|
||||
# — the pre-phase-148 viewer).
|
||||
fallback_fired = False
|
||||
pictorial_described = False
|
||||
# Phase 133 (task 03): the garbage gate — BEFORE the digest/chunk/
|
||||
# summary pipeline, so it runs on the FINAL content exactly once.
|
||||
# A clean layer never reaches the fallback (the gate is the only
|
||||
# gate — no fallback vision calls on the happy path).
|
||||
if not is_usable_pdf_text(content):
|
||||
summary.pdfs_screenshotted += 1
|
||||
fallback_fired = True
|
||||
content = await _fallback_page_screenshots(
|
||||
session, source=source, rel=rel, full_path=full_path, llm=llm
|
||||
)
|
||||
@@ -1457,6 +1506,7 @@ async def _index_pdf_file(
|
||||
source=source, rel=rel, full_path=full_path, llm=llm, summary=summary
|
||||
)
|
||||
if passed is not None:
|
||||
pictorial_described = True
|
||||
content = passed
|
||||
if not content:
|
||||
logger.warning(
|
||||
@@ -1515,6 +1565,35 @@ async def _index_pdf_file(
|
||||
|
||||
session.flush() # guarantees doc.id even for brand-new rows
|
||||
|
||||
# Phase 148 (task 01): the PERSISTENT PAGE COPIES — a qualifying
|
||||
# PDF (the eligibility signals above) gets every page rendered to
|
||||
# ``settings.pdf_page_dir`` (``<doc-id>-p{N}.png``) and
|
||||
# ``documents.page_count`` set. RIGHT AFTER the flush (the doc has
|
||||
# its id — the copy names need it) and BEFORE the chunk pipeline
|
||||
# (the count commits ATOMICALLY with the doc + chunks). Runs ONLY
|
||||
# in this new/changed path: the phase-147 fast path returned
|
||||
# before the branch (an unchanged PDF re-renders NOTHING — the
|
||||
# copies persist) and the content-digest hit above took the shared
|
||||
# unchanged path (no page work there either). FAIL-SOFT: the
|
||||
# render is a DISPLAY concern — any exception leaves
|
||||
# ``page_count`` NULL, the warning is the signal, and the document
|
||||
# proceeds to chunk/embed/summary EXACTLY as pre-phase.
|
||||
try:
|
||||
_persist_pdf_pages(
|
||||
doc,
|
||||
full_path=full_path,
|
||||
page_dir=Path(llm.settings.pdf_page_dir).expanduser(),
|
||||
eligible=fallback_fired or pictorial_described,
|
||||
source=source,
|
||||
rel=rel,
|
||||
)
|
||||
except Exception as e:
|
||||
doc.page_count = None
|
||||
logger.warning(
|
||||
"import: PDF page render failed source=%s path=%s — %s",
|
||||
source, rel, e,
|
||||
)
|
||||
|
||||
# Phase 1+2 — the shared chunk/replace/embed pipeline (the page-
|
||||
# aware chunker dispatches on the ``.pdf`` suffix; empty content
|
||||
# clears the chunk rows and returns 0), committed atomically with
|
||||
@@ -1757,6 +1836,91 @@ async def _pictorial_pdf_pass(
|
||||
return "\n\n".join(parts)
|
||||
|
||||
|
||||
def _persist_pdf_pages(
|
||||
doc: Document,
|
||||
*,
|
||||
full_path: Path,
|
||||
page_dir: Path,
|
||||
eligible: bool,
|
||||
source: str,
|
||||
rel: str,
|
||||
) -> None:
|
||||
"""Phase 148 (task 01): the persistent page copies of one NEW/CHANGED
|
||||
PDF — the display strip's bytes (task 02's public route serves them).
|
||||
|
||||
* ELIGIBLE (the phase-133 fallback fired, or the phase-140
|
||||
pictorial pass described ≥1 page — the sync's own decisions, see
|
||||
the caller's signals): EVERY page of the PDF is rendered (150 dpi,
|
||||
:func:`render_pdf_pages`' default — the phase-133 constant; the
|
||||
``pages=None`` "list index *i* is page *i*+1" contract) and each
|
||||
NON-EMPTY PNG is written to *page_dir* as
|
||||
``<doc-id>-p{N}.png`` (N 1-based), then ``doc.page_count`` is set
|
||||
to the PDF's page count (``len(pages)`` — one slot per page, so
|
||||
the list IS the page count for an openable file). A per-page
|
||||
``b""`` (an unreadable page) is a GAP, not an error: no file is
|
||||
written for it (the route 404s that N, the viewer's note covers
|
||||
it), and the count still covers the full range.
|
||||
* the STALE-REPLACEMENT convention (phase 122's image copies): the
|
||||
doc's EXISTING copies (1..``doc.page_count`` — the row knows its
|
||||
own extent, read BEFORE any change) are deleted FIRST, so a
|
||||
changed PDF's old page set (e.g. 30 pages → 24) cannot leak
|
||||
``p25..p30``; a fresh doc's NULL count is a no-op.
|
||||
* NOT eligible, or an unopenable file (:func:`render_pdf_pages` →
|
||||
``[]``), or a render with ZERO non-empty pages: ``page_count`` is
|
||||
None + the existing copies deleted — a document that STOPPED
|
||||
qualifying loses its strip (e.g. a new version gained a real text
|
||||
layer).
|
||||
* FAIL-SOFT: a write failure removes the copies THIS CALL wrote
|
||||
(no partial-write orphans) and re-raises — the caller
|
||||
(:func:`_index_pdf_file`) catches, leaves ``page_count`` NULL,
|
||||
logs the warning, and the document indexes exactly as pre-phase
|
||||
(the render is a display concern — it never gates indexing).
|
||||
|
||||
*page_dir* is ``Path(settings.pdf_page_dir).expanduser()`` (the
|
||||
``image_dir`` convention — raw string, expanduser'd by the
|
||||
importer); the directory is created on demand, only when a page is
|
||||
actually written (an ineligible doc never creates it).
|
||||
"""
|
||||
# 1 — the stale-replacement: this doc's EXISTING copies go first —
|
||||
# a changed PDF's old page set must not leak, and a doc that
|
||||
# stopped qualifying loses its strip (a fresh doc's NULL count is
|
||||
# a no-op; missing files are a silent no-op).
|
||||
_delete_page_copies(doc, page_dir)
|
||||
if not eligible:
|
||||
doc.page_count = None
|
||||
return
|
||||
pages = render_pdf_pages(full_path)
|
||||
if not pages or not any(pages):
|
||||
# Unopenable file (the render's ``[]``) or every page a
|
||||
# placeholder: no strip — fail-soft, the index is untouched
|
||||
# (the render helper already logged the per-page failures).
|
||||
logger.warning(
|
||||
"import: PDF page render produced no pages source=%s path=%s",
|
||||
source, rel,
|
||||
)
|
||||
doc.page_count = None
|
||||
return
|
||||
page_dir.mkdir(parents=True, exist_ok=True)
|
||||
written: list[Path] = []
|
||||
try:
|
||||
for n, png in enumerate(pages, start=1):
|
||||
if not png:
|
||||
# A per-page render failure: the GAP (no file — the
|
||||
# route 404s that N, the viewer's note covers it).
|
||||
continue
|
||||
path = page_dir / f"{doc.id}-p{n}.png"
|
||||
path.write_bytes(png)
|
||||
written.append(path)
|
||||
except Exception:
|
||||
# A partial write leaves no orphan: the copies this call wrote
|
||||
# go, then the re-raise hands the failure to the caller's
|
||||
# fail-soft handler (page_count NULL + the warning).
|
||||
for path in written:
|
||||
path.unlink(missing_ok=True)
|
||||
raise
|
||||
doc.page_count = len(pages)
|
||||
|
||||
|
||||
async def _index_pdf_embedded_images(
|
||||
session: Session,
|
||||
*,
|
||||
@@ -2800,6 +2964,28 @@ def _delete_image_copy(image_path: str | None) -> None:
|
||||
logger.warning("import: could not delete image copy %s — %s", image_path, e)
|
||||
|
||||
|
||||
def _delete_page_copies(doc: Document, page_dir: Path) -> None:
|
||||
"""Best-effort removal of one document's stale PDF page copies
|
||||
(phase 148, task 01 — the :func:`_delete_image_copy` convention):
|
||||
the ``<doc-id>-p{N}.png`` files for N in 1..``doc.page_count`` —
|
||||
the row knows its own extent (a NULL/zero count is a no-op, which
|
||||
covers fresh docs, non-qualifying docs, and every fail-soft miss).
|
||||
|
||||
A missing file is a silent no-op (already gone — e.g. the owner
|
||||
cleaned the pages dir); an unreadable one is logged, never raised —
|
||||
copy cleanup must not break the sync (the doc row's fate is decided
|
||||
by the upsert/prune logic, not by filesystem hygiene).
|
||||
"""
|
||||
if not doc.page_count:
|
||||
return
|
||||
for n in range(1, doc.page_count + 1):
|
||||
path = page_dir / f"{doc.id}-p{n}.png"
|
||||
try:
|
||||
path.unlink(missing_ok=True)
|
||||
except OSError as e:
|
||||
logger.warning("import: could not delete PDF page copy %s — %s", path, e)
|
||||
|
||||
|
||||
async def _index_image_file(
|
||||
session: Session,
|
||||
*,
|
||||
@@ -3020,6 +3206,7 @@ def _prune(
|
||||
seen: set[tuple[str, str]],
|
||||
images: bool = False,
|
||||
summary: ImportSummary | None = None,
|
||||
page_dir: Path | None = None,
|
||||
) -> int:
|
||||
"""Delete documents of *source_names* whose file is no longer in *seen*.
|
||||
|
||||
@@ -3050,6 +3237,14 @@ def _prune(
|
||||
appended BEFORE the ``session.delete`` — so the sync's incremental
|
||||
folder-summary regeneration (phase 127) sees the pruned docs' folders
|
||||
too. Callers passing nothing keep the pre-phase count-only contract.
|
||||
|
||||
*page_dir* (phase 148, task 01): when given (the sync's
|
||||
``Path(settings.pdf_page_dir).expanduser()``), every pruned doc's
|
||||
PDF page copies (1..``doc.page_count`` — a NULL count is a no-op,
|
||||
covering non-qualifying docs) are deleted beside the row and the
|
||||
image copy — no orphans when a page-qualifying PDF (its parent or
|
||||
a standalone doc) leaves the source. Callers passing nothing keep
|
||||
the pre-phase copy-cleanup contract.
|
||||
"""
|
||||
if not source_names:
|
||||
return 0
|
||||
@@ -3069,6 +3264,11 @@ def _prune(
|
||||
# invisible to the walk, not deleted.
|
||||
continue
|
||||
_delete_image_copy(doc.image_path)
|
||||
# Phase 148 (task 01): the page-qualifying doc's page
|
||||
# copies go with the row (a pruned parent PDF's copies too
|
||||
# — the same site as its embedded-image copies' deletion).
|
||||
if page_dir is not None:
|
||||
_delete_page_copies(doc, page_dir)
|
||||
if summary is not None:
|
||||
summary.changed_paths.append((doc.source, doc.path))
|
||||
session.delete(doc)
|
||||
|
||||
@@ -602,6 +602,19 @@ class DocContent(BaseModel):
|
||||
#: repository's default branch — nothing to show); also dropped by
|
||||
#: the serializer (never ``null``).
|
||||
origin_branch: str | None = None
|
||||
#: Phase 148 (task 02) — the rendered-page BYTES routes for a
|
||||
#: page-QUALIFYING PDF (the phase-133 fallback class or the
|
||||
#: phase-140 pictorial class): ``/api/documents/<id>/pages/<i>``
|
||||
#: for ``i`` in 1..``page_count`` (1-based — the importer's
|
||||
#: ``<doc-id>-p{N}.png`` copy names; a per-page render failure
|
||||
#: leaves a GAP the route 404s — the viewer's note covers it). The
|
||||
#: viewer's page strip (task 03) renders from it; the document's
|
||||
#: text stays in the DB for Q&A, it is simply not displayed. ABSENT
|
||||
#: from the wire for every OTHER document — text docs, images,
|
||||
#: text-heavy PDFs (``None`` → dropped by the serializer — never
|
||||
#: ``null``, the :attr:`image_url` omission precedent), so a
|
||||
#: non-qualifying doc's payload is byte-identical to pre-phase.
|
||||
page_urls: list[str] | None = None
|
||||
|
||||
@model_serializer(mode="wrap")
|
||||
def _serialize(self, handler: SerializerFunctionWrapHandler) -> Any:
|
||||
@@ -612,6 +625,8 @@ class DocContent(BaseModel):
|
||||
data.pop("origin_url", None)
|
||||
if self.origin_branch is None:
|
||||
data.pop("origin_branch", None)
|
||||
if self.page_urls is None:
|
||||
data.pop("page_urls", None)
|
||||
return data
|
||||
|
||||
|
||||
|
||||
Reference in New Issue
Block a user