From a6293f739bc2fbaf54548d1af2b339746b314591 Mon Sep 17 00:00:00 2001 From: ducoterra Date: Fri, 25 Sep 2026 08:37:28 -0400 Subject: [PATCH] docs(readme): document image support (phases 122/123) + full env reference MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit New "Image support" section covering both opt-in capabilities gated by BOR_IMAGES: image documents (chat-model vision description becomes the document's content/summary, only the description is embedded, persistent image_dir copies, Sources thumbnails, viewer + inline chat source rendering, prune guard, fail-soft images_failed counting) and question images (config-gated composer paperclip, server-stored bytes under chat_image_dir, path — never base64 — persisted in saved/shared chats, current-turn-only multimodal message, toggle-off hinted error frame). Intro pointer, import-formats note, and a troubleshooting entry added. The Configuration reference is restructured into grouped tables and now documents EVERY variable in .env.example (was ~34 of 66): app, database pool/concurrency, LLM retries/timeout/thinking, RAG tiers + prompt budgets, chunking/embedding, recency/name-hit ranking, read cap, docs push, image support, and DEBUGPY_PORT. LLM-Generated: true --- README.md | 192 +++++++++++++++++++++++++++++++++++++++++++++++++++--- 1 file changed, 183 insertions(+), 9 deletions(-) diff --git a/README.md b/README.md index ab7fc69..f1f7ad8 100644 --- a/README.md +++ b/README.md @@ -11,6 +11,10 @@ streams a grounded answer back. If it doesn't have notes for your question, it admits it — and offers 2-3 concrete alternative questions about things it **does** have notes on. +Image support is opt-in (`BOR_IMAGES`): standalone images become +retrievable documents, and you can attach an image to a question — +see [Image support](#image-support). + > **Updated your notes?** Re-run the import — it's idempotent and only > re-embeds what changed: > ```bash @@ -200,6 +204,9 @@ reflect the new files. `__pycache__`, etc.) are skipped. - Non-markdown files get format-aware chunking (YAML keys, JSON keys, Python classes/defs). +- With `BOR_IMAGES=1`, standalone images (`png, jpg, jpeg, webp, gif, + bmp`) are indexed as documents too — see + [Image support](#image-support). ### Git-based sources @@ -375,6 +382,78 @@ import summary line (`summaries=N summary_errors=N`). --- +## Image support + +One toggle gates both image capabilities: `BOR_IMAGES` (default +**off**). Enable it **only when your chat model +(`BOR_LLM_CHAT_MODEL`) supports vision** — the chat model is the vision +model: it describes image documents and answers questions with attached +images. The embedding model never sees pixels in either flow. + +### Image documents + +With `BOR_IMAGES=1`, a standalone image file — inside an uploaded +source archive, or as a file in a git/local source walk — becomes a +first-class document: + +- **Described by the chat model.** On first index, a single multimodal + chat-model call describes the image. The description becomes the + document's `content` **and** `summary` — and is the *only* text that + gets embedded, so hybrid search has a natural-language target to hit + instead of pixels. +- **The bytes persist.** Each ingested image is copied to + `BOR_IMAGE_DIR/.` and served from that copy (uploads are + replaced and checkouts re-cloned, so the served image outlives its + source file). +- **Retrieved & displayed.** Image docs appear in the Sources catalog + with a small lazy-loaded thumbnail (progressive enhancement — a fetch + failure falls back to the doc icon), the document viewer renders the + image with the description below it, and a cited image doc renders + **inline in the chat's sources block** (alt text = the description). + The agent's `read` tool returns the description with a marker telling + the model it is reading an image document. +- **Prune guard.** While `BOR_IMAGES` is **off**, syncs never prune + existing image documents — an image doc is invisible to the walk, not + a deleted file. With the toggle on, a deleted image file prunes its + doc and its copy. +- **Fail-soft.** A failed or empty description skips that image (no + row, no copy), logs a warning, and counts in the import summary line + (`images_failed=N`) — the sync continues. + +Accepted formats are `png, jpg, jpeg, webp, gif, bmp` +(`BOR_IMAGE_EXTENSIONS`, case-insensitive) — a separate set from +`BOR_IMPORT_EXTENSIONS`; images are never added via that variable, the +toggle is the single knob. + +### Question images + +With the toggle on, the composer gains a **paperclip** button. Attach +one image to a question and the chat model answers with text **and** +image: + +- **Server-stored, path-persisted.** On send the file is first uploaded + to `POST /api/chat-images` (capped at `BOR_CHAT_IMAGE_MAX_MB`, + default 10 MiB — over is a 413) and stored under `BOR_CHAT_IMAGE_DIR`. + Saved and shared chats carry the returned *path* — never base64. +- **Current turn only.** The model receives a multimodal message for + this question; prior turns' images are never replayed into history + (the text of a prior turn stands alone). +- **Rendered everywhere.** The image renders in the user bubble, after + a page reload, and on the shared chat page; a failed load degrades to + a small "image unavailable" note (never a broken-image icon). +- **Blocked on failure.** If the image upload fails, the send is + blocked with a banner — the question is never sent without the image + the user attached. +- **Toggle off** ⇒ the attach control is absent, and an API request + carrying an image gets an error frame with the hint: *Enable + `BOR_IMAGES` in the server's `.env` (and restart) to ask with an + image.* + +A question image is **never indexed as a document** — it is turn-local +storage, not a source. + +--- + ## Caching & deploys A deploy is a commit — and the browser must see it **without a hard refresh**. @@ -402,40 +481,131 @@ asset references of the known pages in flight. ## Configuration reference +Everything is an environment variable (the `BOR_` prefix, plus +`DEBUGPY*`) read from `.env`. **Every variable in +[`.env.example`](.env.example) is documented below** — copy that file +to `.env` and adjust. + +### App & branding + | Env | Default | Meaning | |-----|---------|---------| | `BOR_APP_NAME` | `Brain of Reese` | Display name everywhere (page titles, header brand, status labels) | | `BOR_INPUT_PLACEHOLDER` | `Ask me anything…` | Chat composer placeholder | | `BOR_FOOTER_TEXT` | `Powered by self-hosted models` | Footer line on every page | +| `BOR_ENVIRONMENT` | `development` | `development` or `production` (the prod compose profile sets `production`) | +| `BOR_LOG_LEVEL` | `INFO` | App log level | +| `BOR_STATIC_DIR` | `frontend` | Where static files are served from (the container image sets `/app/static`) | + +### Database & concurrency + +| Env | Default | Meaning | +|-----|---------|---------| | `BOR_DATABASE_URL` | local compose URL | SQLAlchemy URL (psycopg) | +| `BOR_DB_POOL_SIZE` | `5` | DB connection pool size | +| `BOR_DB_POOL_MAX_OVERFLOW` | `10` | Overflow connections above `BOR_DB_POOL_SIZE` | +| `BOR_DB_POOL_RECYCLE` | `3600` | Recycle pool connections after N seconds | +| `BOR_CHAT_MAX_CONCURRENT` | `10` | Max concurrent `/api/chat` turns | + +### LLM (aipi) + +| Env | Default | Meaning | +|-----|---------|---------| | `BOR_LLM_BASE_URL` | `https://aipi.reeseapps.com/v1` | OpenAI-compatible endpoint | | `BOR_LLM_API_KEY` | — (falls back to `$AIPI_KEY`) | aipi API key | -| `BOR_LLM_CHAT_MODEL` | `turbo` | Chat model | +| `BOR_LLM_CHAT_MODEL` | `turbo` | Chat model — must support vision when `BOR_IMAGES=1` | | `BOR_LLM_EMBED_MODEL` | `embed` | Embedding model | | `BOR_LLM_SUMMARY_MODEL` | `lite` | One-shot completions: document summaries at import, KB overview | | `BOR_EMBEDDING_DIM` | `768` | Vector dimension (fixed at table creation) | -| `BOR_TOP_N_DOCS` | `2` | Full documents fed to the LLM | +| `BOR_LLM_RETRIES` | `3` | Retries: chat-turn stream before the first token + empty one-shot summary replies; `0` = off | +| `BOR_LLM_RETRY_DELAY` | `5` | Seconds between LLM retries | +| `BOR_LLM_TIMEOUT` | `120` | HTTP timeout (s) for LLM API calls | +| `BOR_STREAM_THINKING` | `1` | Stream the model's thinking as `thinking` SSE events (`0` = suppress) | + +### RAG & prompt budgets + +| Env | Default | Meaning | +|-----|---------|---------| +| `BOR_TOP_N_DOCS` | `2` | Ceiling for the cited tier, not a quota (phase 113) — a single strong doc yields one citation | | `BOR_RELEVANCE_THRESHOLD` | `0.62` | Answer when best cosine ≥ this **or** a lexical hit corroborated by cosine ≥ `BOR_LEXICAL_SUPPORT_FLOOR`; otherwise honest deflection | | `BOR_LEXICAL_SUPPORT_FLOOR` | `0.35` | Best-cosine floor an FTS hit must clear to flip the gate HIGH (A8 revised 2026-09-14) | +| `BOR_SOURCE_USEFULNESS_FLOOR` | `0.35` | Cosine bar for a citation slot (phase 113): a doc earns the cited tier only when its best-chunk cosine clears it; `0` = no bar | +| `BOR_RELATED_MAX_DOCS` | `2` | Cap on the related-doc tier (the de-emphasized "nearby docs" row — never a citation chip); `0` = no related docs | +| `BOR_SUGGESTED_DOCS` | `5` | Cap on the "start here" suggestion tier (phase 118): top-N doc summaries seeded into the grounded prompt; no cosine floor; ≥ 1 | +| `BOR_SUGGESTION_PREVIEW_CHARS` | `400` | NULL-summary fallback preview in a suggestion block (phase 118): first N content chars + truncation marker; > 0 | +| `BOR_MAX_OUTPUT_TOKENS` | `32768` | Max answer length in tokens (answers must not be cut off) | +| `BOR_STEERING_MAX_CHARS` | `8000` | Char budget for the `` (steering notes) prompt section | +| `BOR_SUMMARY_MAX_CHARS` | `12000` | Cap on document content sent to the `lite` summary model | +| `BOR_KB_OVERVIEW_MAX_CHARS` | `4000` | Char budget for the `` prompt section | +| `BOR_OVERVIEW_INPUT_MAX_CHARS` | `40000` | Cap on the document list sent to `lite` for KB overview generation | +| `BOR_FOLDER_SUMMARY_INPUT_MAX_CHARS` | `8000` | Cap on a folder's document list sent to `lite` for folder summaries (phase 94) | + +### Chunking & embedding + +| Env | Default | Meaning | +|-----|---------|---------| +| `BOR_CHUNK_TARGET_CHARS` | `2000` | Target chunk size (chars) | +| `BOR_CHUNK_OVERLAP_CHARS` | `200` | Overlap between adjacent chunks (chars) | +| `BOR_EMBED_BATCH_SIZE` | `16` | Chunks per embedding API call | +| `BOR_EMBED_QUESTION_MAX_CHARS` | `1200` | Chat question-embed prefix cap (phase 114): the embed step embeds at most this many chars of the question — the full question still reaches the LLM prompt | + +### Hybrid retrieval + +| Env | Default | Meaning | +|-----|---------|---------| | `BOR_HYBRID_VECTOR_CANDIDATES` | `100` | Cosine list width for the RRF fusion | | `BOR_HYBRID_LEXICAL_CANDIDATES` | `30` | FTS list width for the RRF fusion | | `BOR_RRF_K` | `60` | RRF damping constant (`1/(k + rank)`) | +| `BOR_RECENCY_BOOST` | `0.0007` | Max additive fused-score bonus a zero-age doc gets (phase 106); `0` = off | +| `BOR_RECENCY_HALF_LIFE_DAYS` | `365` | Recency decay timescale, days (phase 106) — the boost decays by e^-1 (~0.37x) per timescale of age | +| `BOR_NAME_HIT_BONUS` | `0.005` | Selection-time bonus a name-hit doc's best fused score gets (phase 119); `0` = off | + +### Agent tools + +| Env | Default | Meaning | +|-----|---------|---------| | `BOR_AGENT_MAX_ROUNDS` | `10` | Hard cap on agent tool rounds per grounded turn (0 = no tools) | -| `BOR_IMPORT_EXTENSIONS` | csv (see below) | Importable formats (may only narrow the A9 set) | +| `BOR_READ_MAX_CHARS` | `128000` | Cap on a `read` tool result (chars); over-cap reads truncate + a grep-pointer notice (phase 95) | + +### Import & sources + +| Env | Default | Meaning | +|-----|---------|---------| +| `BOR_IMPORT_EXTENSIONS` | A9 family | Importable text formats — the A9 default is `md,markdown,txt,yaml,yml,json,py,container,network,volume,image,pod,kube,swap,os,endpoint,j2`; any well-formed extension is allowed. Images are **not** added here — see `BOR_IMAGES` | | `BOR_GIT_SOURCES` | — (empty) | CSV of git repo URLs — **fallback while the admin Git sources page's list is empty** | | `BOR_SOURCES_DIR` | `~/bor-sources` | Where git repos are cloned/pulled | | `BOR_UPLOAD_DIR` | `~/bor-sources/uploads` | Where uploaded source archives are unpacked | | `BOR_UPLOAD_MAX_MB` | `512` | Cap (MiB) for uploaded source archives (zip-bomb guard) | -| `BOR_STEERING_MAX_CHARS` | `8000` | Char budget for the `` prompt section | -| `BOR_SUMMARY_MAX_CHARS` | `12000` | Cap on document content sent to the `lite` summary model | -| `BOR_KB_OVERVIEW_MAX_CHARS` | `4000` | Char budget for the `` prompt section | -| `BOR_OVERVIEW_INPUT_MAX_CHARS` | `40000` | Cap on the document list sent to `lite` for KB overview generation | | `BOR_SUGGESTIONS` | built-in list | JSON seed for onboarding chips (shown before the first saved question) | + +### Image support (phases 122/123) + +| Env | Default | Meaning | +|-----|---------|---------| +| `BOR_IMAGES` | `0` | Master switch for image support — **off by default; enable only when `BOR_LLM_CHAT_MODEL` supports vision**. Gates both image documents and question images | +| `BOR_IMAGE_EXTENSIONS` | `png,jpg,jpeg,webp,gif,bmp` | Extensions indexed as image documents (case-insensitive; separate from `BOR_IMPORT_EXTENSIONS`) | +| `BOR_IMAGE_DIR` | `~/bor-sources/images` | Persistent home for the served image bytes (uploads are replaced, checkouts re-cloned) | +| `BOR_CHAT_IMAGE_DIR` | `~/bor-sources/chat-images` | Where question images are stored (saved/shared chats carry the path, never base64) | +| `BOR_CHAT_IMAGE_MAX_MB` | `10` | Question-image upload cap (MiB; must be > 0) | + +### Docs push + +| Env | Default | Meaning | +|-----|---------|---------| +| `BOR_DOCS_REPO` | — (empty) | The git repo chat answers can be committed to (URL or local path); while empty "Save as doc" is hidden and the push endpoint 409s | +| `BOR_DOCS_BRANCH` | `bor-docs` | Branch pushes land on (`git push --ff-only` — you open the PR yourself) | +| `BOR_DOCS_BASE_BRANCH` | `main` | Branch each push is cut fresh from | +| `BOR_DOCS_WORK_DIR` | `~/bor-docs` | Where `BOR_DOCS_REPO` is checked out on the server | + +### Admin & debugging + +| Env | Default | Meaning | +|-----|---------|---------| | `BOR_ADMIN_PASSWORD` | *(required)* | The single admin's password — app refuses to start when empty | | `BOR_SESSION_SECRET` | *(required)* | Signing key for the `bor_session` cookie | | `BOR_SESSION_MAX_AGE` | `43200` | Session-cookie lifetime in seconds (12 h, sliding) | -| `DEBUGPY` | `0` | `1` ⇒ attach-on-demand debugpy on `DEBUGPY_PORT` (default 5678) | -| `BOR_LOG_LEVEL` | `INFO` | App log level | +| `DEBUGPY` | `0` | `1` ⇒ attach-on-demand debugpy on `DEBUGPY_PORT` | +| `DEBUGPY_PORT` | `5678` | debugpy listen port | ### Customizing the look @@ -576,6 +746,10 @@ served locally (no CDN), `BOR_ENVIRONMENT=production`. - **Embedding dimension mismatch** — aipi changed models; run `uv run python -m scripts.llm_probe`, update `BOR_EMBEDDING_DIM`, then drop + recreate the chunks table. +- **No attach button / images ignored on import** — image support is + off (the default). Set `BOR_IMAGES=1` in `.env` and restart — only + when `BOR_LLM_CHAT_MODEL` supports vision. Turning the toggle off + never destroys existing image documents (syncs prune-guard them). - **Honest deflection** (the amber deflection bubble) — every question passes the honesty gate: deflection happens when the best cosine similarity is below `BOR_RELEVANCE_THRESHOLD`