phase: 148_pdf_pages_in_sources
Build and Push Containers / build-and-push-app (push) Successful in 2m28s
Build and Push Containers / build-and-push-db (push) Successful in 16s

**Phase 148 — final verification pass: complete, 100% green.**

- **Verified (tasks 01–04 already implemented):** page render/persist at index (`_persist_page_copies`, fail-soft, stale-replace, qualifier-loss, prune hook via `_delete_page_copies`); public `GET /api/documents/{id}/pages/{n}` (uuid4-credential, 404-all-non-servable, `image/png` + cache header); `DocContent.page_urls` additive-omitted; shared `renderDocument` strip (lazy imgs, "Page N" captions, text + Summary suppressed, onerror note, createElement-only); migration 0026; `pdf_page_dir` setting; `.env.example` entry
- **Completion criteria:** (1) pictorial + fallback classes get `p{N}.png` + `page_count`, text-heavy gets neither — pinned, pass; (2) unchanged re-sync renders zero (fast path) — pass; (3) changed re-index replaces copies, qualifier-loss/prune delete — pass; (4) route 404 matrix + payload key absent-never-null — pass; (5) modal AND /document.html strip with suppression, text-heavy byte-identical — E2E pass; (6) see below; (7) no completed-phase drift — full suite green
- **Tests:** `uv run pytest --cov=app --cov-report=term-missing` → **3461 passed, 0 failed**, `app/` coverage **99%** (>90%); `uv run pytest tests/e2e/test_pdf_pages_in_sources.py -v --no-cov` → **4 passed** in isolation; `uv run ruff check .` clean; `uv run pyright` → 0 errors, 0 warnings
- **Documentation:** README had **1 pre-existing inconsistency** — the Production Deployment section's standalone `podman run` example (and compose note) omitted the **required** `BOR_ADMIN_PASSWORD`/`BOR_SESSION_SECRET` (the fail-loud boot gate the README's own Admin section documents — the container would refuse to start); fixed with the two `-e` flags + a note for the compose path. All else verified consistent: 70/70 `.env.example` vars documented, all defaults match `config.py`, pages table matches routes, Containerfile/esbuild/uid-10001/healthcheck/entrypoint claims accurate, QA commands match pytest config. **AGENTS.md: verified consistent — no changes needed**
- **Notable:** repo uses `Containerfile` (Podman) — README's `podman build` example is valid as-is; no live infrastructure touched
- **Next pending phase:** `149_sync_progress_box`
This commit is contained in:
2026-10-01 16:58:01 -04:00
parent 863519e980
commit b233c8d714
39 changed files with 3426 additions and 56 deletions
+25 -2
View File
@@ -678,8 +678,21 @@ uploaded archives):
pages on the import summary line (`pdfs_pictorial=N`
`pdf_pictorial_pages=N`). `BOR_PDF_PICTORIAL=0` turns the whole pass
off (byte-identical pre-phase-140 content).
- **Served as text.** The document viewer renders the extracted text —
never the binary PDF.
- **Served as text — or as pages (phase 148).** The document viewer
renders the extracted text — never the binary PDF. The exception is
the two page-qualifying classes: a PDF whose text layer fired the
phase-133 screenshot fallback, or a PDF whose phase-140 pictorial
pass described ≥ 1 sparse page. For those, every page is rendered
to a persistent PNG at index time (under `BOR_PDF_PAGE_DIR`) and
the viewer — the doc modal AND `/document.html`, one shared core —
shows the PAGES: a vertical strip of lazy page images with
"Page N" captions, in place of the text content and the Summary
panel (the per-page descriptions stay in the DB for Q&A — the
embeddings and summary are untouched, just not displayed). A lost
copy shows a "Page N unavailable" note for that page only. Text-
heavy PDFs (usable layer, zero pictorial candidates — and the whole
sparse class while `BOR_PDF_PICTORIAL=0`) keep the text view,
byte-identical to pre-phase.
---
@@ -805,6 +818,7 @@ to `.env` and adjust.
| `BOR_IMPORT_EXTENSIONS` | A9 family | Importable text formats — the A9 default is `md,markdown,txt,yaml,yml,json,py,container,network,volume,image,pod,kube,swap,os,endpoint,j2`; any well-formed extension is allowed. Add the `pdf` token to make PDFs first-class, retrievable documents (phase 126, opt-in). Images are **not** added here — see `BOR_IMAGES` |
| `BOR_PDF_PICTORIAL` | `1` | Master switch for the pictorial PDF pass (phase 140, on by default): a usable-text PDF's sparse pages (stripped text under `BOR_PDF_PICTORIAL_MIN_CHARS`) are rendered + vision-described into the indexed content (`[Page N illustration: …]`); `0` ⇒ content byte-identical to pre-phase-140 (no render, no vision call) |
| `BOR_PDF_PICTORIAL_MIN_CHARS` | `100` | Sparseness bar for the pictorial pass (phase 140): a page whose STRIPPED text is STRICTLY shorter than this many chars is a candidate (a page with exactly this many is not); `0` = every page (A21); a negative value fails startup loudly |
| `BOR_PDF_PAGE_DIR` | `~/bor-sources/pages` | Persistent home for the rendered PDF page PNGs (phase 148): a page-qualifying PDF (the phase-133 screenshot class or the phase-140 pictorial class) gets every page rendered at index time as `<doc-id>-p{N}.png` here, so the document viewer shows the actual pages (the descriptions stay embedded for Q&A, just not displayed); text-heavy PDFs keep the text view |
| `BOR_GIT_SOURCES` | — (empty) | CSV of git repo URLs — **fallback while the admin Git sources page's list is empty** |
| `BOR_SOURCES_DIR` | `~/bor-sources` | Where git repos are cloned/pulled |
| `BOR_UPLOAD_DIR` | `~/bor-sources/uploads` | Where uploaded source archives are unpacked |
@@ -954,15 +968,24 @@ podman run -d --name brain-of-reese \
-e BOR_DATABASE_URL=postgresql+psycopg://reese:SECRETPASSWORD@dbhost:5432/brain_of_reese \
-e BOR_LLM_BASE_URL=https://aipi.reeseapps.com/v1 \
-e BOR_LLM_API_KEY=$AIPI_KEY \
-e BOR_ADMIN_PASSWORD=your-password \
-e BOR_SESSION_SECRET=<the-hex-from-the-setup-above> \
brain-of-reese/app:latest
```
`BOR_ADMIN_PASSWORD` and `BOR_SESSION_SECRET` are **required** — the
fail-loud boot gate in [Admin & Sign-in](#admin--sign-in) refuses to
start while either is empty.
The entrypoint runs `alembic upgrade head` automatically on start.
Or run the whole stack from compose (app + db):
```bash
podman compose --profile prod up -d --build
```
The `app` service picks up `BOR_ADMIN_PASSWORD` / `BOR_SESSION_SECRET`
from the shell environment (or an env file) — the same fail-loud gate
applies.
Production hardening: app runs as non-root (uid 10001), slim image,
healthcheck on `/api/health`, debugpy off unless `DEBUGPY=1`, all assets