brain-of-reese/README.md

# 🧠 Brain of Reese

A chippy, honest **RAG chatbot** over your homelab documentation — git
repositories, local directories, and uploaded archives registered on the
admin **Sources** page. Ask it anything about your notes — markdown,
YAML, JSON, Python, plain text, and more — and it retrieves the relevant
chunks with **hybrid search** (pgvector cosine ∪ Postgres full-text
search, fused with Reciprocal Rank Fusion), feeds the **whole relevant
document** to a **self-hosted LLM** (`turbo` via
`https://aipi.reeseapps.com/v1`), and streams a grounded answer back.

If it doesn't have notes for your question, it admits it — and offers
2-3 concrete alternative questions about things it **does** have notes on.

Image support is opt-in (`BOR_IMAGES`): standalone images become
retrievable documents, and you can attach an image to a question —
see [Image support](#image-support).

> **Updated your notes?** Re-run the import — it's idempotent and only
> re-embeds what changed:
> ```bash
> uv run python -m scripts.import_docs --prune
> ```
> See [Updating the documents](#updating-the-documents) for details.

---

## Quick Start

```bash
# 1. Dependencies
uv sync

# 2. Configure (copy and edit)
cp .env.example .env
# Set BOR_LLM_API_KEY — your aipi key (falls back to $AIPI_KEY)
# Set BOR_ADMIN_PASSWORD and BOR_SESSION_SECRET — both are REQUIRED:
# the app refuses to start while either is empty (see Admin & Sign-in)

# 3. Start the database
podman compose up -d db
podman compose ps          # wait until "healthy"

# 4. Apply migrations & import
uv run alembic upgrade head
uv run python -m scripts.import_docs

# 5. Run the app
uv run uvicorn app.main:app --reload
# → http://localhost:8000  (chat)
```

**Stack:** FastAPI · Pydantic v2 · SQLAlchemy 2 · Alembic · pgvector ·
vanilla HTML/CSS/JS (no CDN) · Playwright E2E

---

## Pages at a glance

The app is a single-page application — all views load from `/` with a
client-side router. Direct URLs deep-link to the matching view.

| Page | URL | Who can see it |
|------|-----|----------------|
| **Chat** | `/` | Everyone (token gate for non-shared) |
| **RAG / Knowledge base** | `/sources.html` | Admin only |
| **Git sources** | `/git-sources.html` | Admin only |
| **Tuning** | `/tuning.html` | Admin only |
| **Saved chats** | `/history.html` | Admin only |
| **Access tokens** | `/tokens.html` | Admin only |
| **Theme** | `/theme.html` | Admin only |
| **Settings** | `/settings.html` | Admin only |
| **Document viewer** | `/document.html?source=X&path=Y` | Token users & admins |
| **Login** | `/login.html` | Everyone |
| **Shared chat** | `/shared/<token>` | Everyone (anonymous) |
| **Edit doc** | `/doc-edit.html` | Admin only |

### Chat (`/`)

Ask questions; answers stream in with **source chips** that cite the exact
documents used (a cited image document gets no chip — its compact
figure is the citation, phase 130). Clicking a chip or figure opens the
document in an **almost-fullscreen modal** on the same page (no new tab);
the browser **Back** button closes the modal — instead of navigating the
view behind it — and Forward re-opens the same document (phase 144). The
modal's **Full page** link opens the dedicated `/document.html` viewer in a
new tab carrying its origin: a document opened from Chat returns through
the viewer's back link to Chat, one opened from Sources returns to Sources
(phase 153, feeding the phase-13 `back` contract on every path). The
onboarding suggestion chips
follow the opening questions of the 3 most recent saved chats (each
chat's first question — a follow-up never chips, phase 103) — on a
fresh deployment they seed from `BOR_SUGGESTIONS`. A long opening
question caps at 90 displayed characters with a `…` (phase 157) — the
full text stays in the hover tooltip, and clicking the chip submits
the full question.

**No empty answer box mid-turn** (phase 166): the bubble that will hold the
answer is created on the turn's first `thinking`/`tool` frame — it is the
DOM anchor the Reasoning container inserts itself above — but it stays
hidden until the **first answer token** lands in it, so an in-flight turn
shows the typing indicator and the Reasoning bubble only, never a padded
empty rectangle (nothing replaces the box: the existing loading feedback
is the turn's visible state). The same rule covers a restored or shared
answer whose persisted text is empty, which paints no box either.

**Markdown links render as links** (phase 180, TODO.md L4): an inline
`[text](url)` in an answer, the thinking block, a saved/shared chat, the
document viewer or the document modal — every surface that goes through
the one shared renderer — becomes a real `<a class="md-link">`. Only
allowlisted URLs become live elements: `http(s)://` (which also carries
`target="_blank" rel="noopener noreferrer"`), `mailto:`, `/`-rooted and
`./`/`../` relative paths, and `#fragment` links — the last three stay in
the same tab and carry no `target`. Anything else (`javascript:`,
`data:`, `vbscript:`, `file:`, `blob:`) renders as the plain
`[text](url)` text it is today, never an element. The URL is used
verbatim from the already-escaped text, so a `&` in a query string stays
`&amp;` and nothing is ever unescaped (escape-first is the security
boundary); a URL containing a space or a literal `)` is a documented miss.
Styling is one `.md-link` rule for every surface: underlined (a link never
relies on colour alone) in the theme's AA-safe brand text colour — 9.9:1 on
the answer bubble, 8.3:1 on a deflected one — and the user bubble's own
ink on the brand fill (5.2:1), where the underline is what marks the link;
Tab focus is the app's global 3px `--brand` ring.

**Markdown images render as images** (phase 180, the in-answer half of
TODO.md L3): an inline `![alt](src)` — and a linked image
`[![alt](src)](url)`, which renders as that image wrapped in the link —
becomes a lazy `<img class="md-img" loading="lazy">` on every surface that
shares the renderer, so a diagram referenced inside an answer or a
document paints instead of printing its reference text. `alt` is always
emitted (empty allowed for a decorative image). The image allowlist is
its own function and accepts `/`-rooted served paths
(`/api/documents/<id>/image`, `/assets/…`), relative paths whose first
segment is at most two characters (`./x`, `../x`, `d/x` — a longer first
segment like `images/x.png` is a documented miss), `http(s)://` and
`data:image/…` — the `data:` case needs no CSP change
because the app's own `img-src 'self' data:` already permits it. A
`javascript:`, `file:`, `blob:` or non-image `data:` src stays visible as
the inert `![alt](src)` text it is today, never an element, and a broken
served path degrades to the browser's alt text (the renderer emits an HTML
string, so it carries no inline `onerror` handler — the JS fallbacks live
in `document.js`/`app.js`). `.md-img` carries the app's bordered-image card
treatment and is capped to its container — `max-width: 100%`, `height: auto`
and `object-fit: contain` — so a wide screenshot never produces a horizontal
scrollbar at 375px or at 1440px. An SVG document that declares no `width`
or `height` is sized by phase 181's `.doc-image-img--vector` class in the
document viewer instead (see "Image documents"); an inline answer image
keeps the container cap above.

**The heading ladder reaches six levels** (phase 180, TODO.md L5): a
`#### x` line in an answer, the thinking block, a saved/shared chat, the
document viewer or the modal renders `<h5>x</h5>`, and `##### x` /
`###### x` render `<h6>x</h6>` (HTML stops at `h6`). The levels above are
UNCHANGED and deliberately flattened — `#`/`##` both render `<h3>`, `###`
renders `<h4>` — so an answer's deep sections gain the two rungs below
`###` without re-ranking what already rendered. Like every other inline
rule the ladder needs a space after its hashes and sits at column 0 of its
line, so a mid-sentence `####` and an indented `#### x` stay plain text,
and a `####` inside a fenced block stays code. The two new rungs are sized
into the existing type scale — `h3` 1.15rem → `h4` 1rem → `h5` 0.95rem →
`h6` 0.88rem, margins tightening with depth — so a deep heading neither
collapses into body text (the UA default is 0.83em/0.67em) nor out-shouts
the answer around it, in the bubble, the thinking block and the viewer
alike.

**The renderer has one more pass, and it is opt-in** (phase 182):
`renderMarkdown(md)` is now `renderMarkdown(md, opts)`, and the ONLY truthy
option is `{ verbatim: true }`, passed by exactly the FOUR answer renderers —
the live per-delta write, the `done`-time re-render, the chat page's restore
and the shared page's record renderer. Every other call site omits it, so the
feature is inert elsewhere. With it
on, a ` ```bor <source/path> <range> ` locator fence is
replaced — fence line, body and closer included — by the pending verbatim
panel, so the model's re-typed copy of a file is never painted (see
[Verbatim text display](#verbatim-text-display-phase-182)). Pass order with
the option on is **verbatim → fences → tables → escape+inline → blockquotes →
paragraphs → restore**; with it off — the document viewer, the thinking runs,
and every question bubble on both pages — the output is byte-identical
to pre-phase, so a document that legitimately contains a ` ```bor ` fence
still renders its own text.

The document's meta row (the chat modal **and** the standalone viewer at
`/document.html`) carries a **Copy path** button: one click copies the
document's `source/path` (the exact form the `read` tool, the chips, and
the RAG table use) to the clipboard; on a non-secure (http) homelab
origin the browser rejects the clipboard, so the click instead offers the
full path in an inline fallback field that selects itself on focus —
click or Tab, then copy (phase 153).

**Edit** the newest question (phase 159, re-housed in phase 170):
the latest user bubble carries an **Edit** button that opens the
editor *inside* the bubble — the editor inherits the bubble's own
box, background, and padding, so it keeps the bubble's dimensions
(the bubble stays visible; the rendered question is replaced by the
form). A labelled textarea is prefilled with the question (the
4,000-character cap is enforced with `maxlength`; **Ctrl/⌘+Enter**
submits, **Esc** / **Cancel** restore the rendered question
byte-identical, and a whitespace-only question is rejected inline).
**Send** rewrites the question in the chat and the saved chat in
place, drops every turn that followed it, and re-asks the trimmed
question — a fresh answer streams in the old answer's place (the
question is never duplicated; an attachment on the question survives
the edit and the re-ask stays text-only). A turn that failed with
"Question too long — trim it and re-ask" (or a mistyped question)
is fixed by trimming it in the editor instead of retyping it.
Earlier bubbles never carry the button (last-question-only scope,
owner-locked).

**When a long answer spends its research budget** (phase 160): the
answer's meta row carries a **Continue researching** button (row order
Tune → Retry → Continue → Save-as-doc; not admin-gated — chat is
public, so any visitor can keep a long research session going). Each
tool result tells the model how many tool calls remain; calls
attempted after the budget is spent are **rejected** (not executed,
not shown) instead of being dumped into the chat, and the turn ends
with a final no-tools reply — the model is coached to answer from the
results it already has and to tell you it can keep going, and a
deterministic "budget spent" fallback line stands in if that reply
carries no visible content. Clicking the button sends one plain
`continue` turn — the budget is per-turn, so the new turn has a whole
new round budget, and nothing server-side knows or cares that this is
a continuation (A10, stateless). The button is inert while a turn is
streaming and single-shot (a stale button is removed when the new
turn starts); the flag is persisted on the saved record, so the button
restores on a refresh — but a shared chat never shows it.

### RAG / Knowledge base (`/sources.html`)

The indexed document list. The *Path* column opens each document in the same
**almost-fullscreen modal** (and `/document.html` renders it through the one
shared core). Content renders by kind (phase 165): markdown becomes a
rendered document, an image's description or a plain-text (`.txt`) body
renders as **wrapping prose** in the 72rem reading column, and code and
code-ish formats keep the monospace block with a horizontal scroller
(long lines never wrap in code — locked; the themed modal scrollbar of
phase 116 keeps applying to whatever still scrolls). Document rows carry
a *Created* date and folder
rows an *Updated* date (phase 106) — sourced at sync time (git commit dates
for git sources, restored metadata for archive uploads, file mtime otherwise;
unresolvable or future dates fall back to today) — and the admin can correct a
document's date from the document viewer. The date also feeds the small
recency boost in retrieval ranking (`BOR_RECENCY_BOOST`). Admin-only —
anonymous visitors see a sign-in gate instead.

The tree payload is bounded (phase 167): `GET /api/docs/tree` builds only
`depth` levels (default **1** — the sources plus one level, the page's first
paint; `depth=2..8` builds deeper, anything else is a 422) instead of the old
~2 MB full recursive dump. A node whose children were not built carries
`"truncated": true` and ships with its **recursive** document count, updated
date, description and pending marker (`children: []`); untruncated nodes are
byte-for-byte what they were before. The three stat cards read the
server-computed top-level `totals` (`documents`, `chunks`, `last_indexed`) so
they stay honest whatever depth was requested. Responses also ship gzipped
(see *Caching & deploys*). Deeper levels load on demand through
`GET /api/docs/tree/node?source=X&path=Y&depth=N` (admin-only, phase 167):
one node — the source root for `path=""` — answered in exactly the shape the
full tree produces for it (unknown source or folder → 404 `unknown node`),
built from the same single catalogue query. The RAG view fetches each drilled
level once and caches it (re-opening a folder costs no network), showing one
*Loading…* row while a level arrives and a *Couldn't load … — Retry* line on
failure with the current level staying rendered; a folder that vanished since
it was listed (the node 404) resets the view to the top level. The
breadcrumb, the bounded scroll panel, and the description editors behave
exactly as before. All three catalogue reads
(`/api/docs`, `/api/docs/tree`, `/api/docs/tree/node`) also carry a weak
`ETag` + `Cache-Control: private, must-revalidate`, so reloading a page you
just looked at costs a bodiless 304 (see *Caching & deploys*).

### Git sources (`/git-sources.html`)

The admin-managed source registry: add or remove git repositories, upload
source archives (`.tar`, `.tar.gz`, `.tgz`, `.zip`), and register local
directories — all in one table with a `kind` discriminator. No `.env`
editing, no restart. The **Sync sources** button — which lives on the
**RAG** page (`/sources.html`), not here — clones/pulls/walks every
source and re-imports in one click.

### Tuning (`/tuning.html`)

Set global instructions that are injected into the system prompt of every
chat turn. Add notes like *"be more concise"* or *"assume I'm on NixOS"*.
List, edit, or remove them here — no conversation required.

### Saved chats (`/history.html`)

Every conversation is saved automatically. Click a title to return to that
chat. Admin-only view. The list is paginated server-side (25 rows per
page) — with more than 25, a Prev / “Page N of M” / Next pager appears
under the table to walk it, and **Refresh** re-fetches the current page.
`GET /api/chats` carries the same contract: `?limit=` (default 25,
clamped 1–100) / `?offset=` and a full `total` on every response.

### Access tokens (`/tokens.html`)

Generate tokens and hand them out — a token opens chat, the answers, and
the documents they cite. Shared chats stay open to everyone.

### Settings (`/settings.html`)

Edit any setting that today lives in `.env` — every `BOR_*` setting, grouped
and type-correct — from the browser. A value you **Save** is stored in the
database and **wins over the environment variable from that instant on, with
no restart** (boot-wired settings — the pool, the session middleware, the log
level — carry a `restart required` badge and apply on the next start); **Reset**
hands control back to the environment, and **Clear all overrides** empties the
override table. `.env` is never written: the environment stays the base, the
database row wins. Secrets are write-only (you set a value, never read one
back); a value that fails validation is rejected before anything is stored. 
Admin-only — anonymous visitors see a sign-in gate.

The overlay can be switched off for the whole process with
`BOR_SETTINGS_DB=0` (env-only; the panel stays visible and says so).

### Shared chat (`/shared/<token>`)

A read-only snapshot of any conversation, shareable via a link. Open to
everyone — no login or token required. The page still ships no composer, no
Save/Share/Tune/Retry and no chips-as-buttons; the ONE interactive exception is
phase 182's answer code: a shared verbatim panel and the answer's ordinary
code blocks carry a **Copy** button and the panel a plain
`/document.html?source=…&path=…` link, because a page that shows a slice of a
file has to let the reader copy it and check it. Both come from the same
`verbatim.js` the chat page calls (so a shared turn renders identically to the
owner's), and a guest who follows the viewer link lands on the sign-in page —
the documents API is admin-gated.

### Conversation payload limits

A conversation has one shape, and it carries the **same static bounds**
whether it rides back to the model as `history` on the next turn or is
written to a saved chat — the two paths mirror each other on purpose, so
saving a conversation can never be stricter than continuing it (phase
164):

| Bound | Value | What it admits |
|-------|-------|----------------|
| One message (`text`, `thinking`) | 600 000 chars | a **full-length answer at the largest answer cap** — `BOR_MAX_OUTPUT_TOKENS` at 131 072 tokens ≈ 524 300 chars — as history and as a saved record, with margin (the old 32 000 ceiling broke even at the 32 768-token default) |
| Turns sent as `history` | 500 | the whole browser conversation (the client sends everything it holds, minus the question being asked) |
| Messages per saved chat | 500 | the same conversation, saved |
| One verbatim block (`answer_text[].text`) | 600 000 chars | the same answer-text cap (phase 182) — comfortably above `BOR_SHOW_TEXT_MAX_CHARS`, so raising that budget never hits the wire limit |
| Verbatim blocks per saved message | 50 | the `tools` / `segments` number (phase 182) — far past the `BOR_SHOW_TEXT_MAX_BLOCKS` default of 5 |
| **Total chars per saved chat** | **8 000 000** | ~15 full-length answers; the one bound on the anonymous save surface, which stays bounded on purpose — and it counts the verbatim blocks' bytes too (phase 182), so the display blocks cannot become a second, unbounded byte channel |
| One question (`message`) | 4 000 chars | unchanged; the composer counts it live |

These are sanity ceilings, not the prompt's budget: the real gate is
`BOR_HISTORY_MAX_TURNS` / `BOR_HISTORY_MAX_CHARS`, and a turn that does
not fit the budget is dropped **whole** from later prompts (never cut
mid-answer). So a very long answer stops being context for the next
question — the follow-up still works, just with a trimmed window —
raise `BOR_HISTORY_MAX_CHARS` if you want it kept.

**A failed turn states what actually happened** (phase 164): when the
app *answers* but rejects the send before the model runs (an HTTP
4xx/5xx — a validation rejection, an outage page, a proxy error), the
bubble reads *"I couldn't take that — my API rejected the request. Use
Retry to ask again."* — not the connection line — and the banner adds
that nothing was lost but that send. A genuine transport failure and a
model-side error keep *"My answer didn't make it — the connection
dropped."* Either way the exact reason rides verbatim on the bubble's
**Failed** note (API validation details serialize as their `msg`
text — never `[object Object]`, and rejected request content is never
shown), persists with the record, and lands Retry + Edit.

### Edit doc (`/doc-edit.html`)

Review and adjust an AI-generated answer before committing it to the docs
repository. A draft that should never be pushed can be **Discard**ed from
the same page (permanent; admin-only). Admin-only flow page.

---

## Admin & Sign-in

Brain of Reese has exactly **one account: the admin (you)**. Signing in
unlocks the **full RAG catalog** (`/sources.html`), the **answer-tuning**
controls, and **token management**. **Shared chats** are the only content
that stays open to anonymous visitors.

### Setup (one-time)

```bash
python -c 'import secrets;print(secrets.token_hex(32))'   # → paste into .env
```

```env
BOR_ADMIN_PASSWORD=your-password        # plaintext — homelab scope, by design
BOR_SESSION_SECRET=<the hex from above> # signs the session cookie
```

**Fail-loud:** while either variable is empty the app refuses to start,
naming the missing one(s).

### How authentication works

| Endpoint | Purpose |
|----------|---------|
| `POST /api/login` | Password → signed `bor_session` cookie |
| `POST /api/logout` | Clears session cookie |
| `GET /api/whoami` | `{"authenticated": bool, "role": "admin"|"user"|"anonymous"}` |
| `POST /api/token-auth` | Token (`bor_…`) → same cookie, role `user` |

The cookie uses `same_site="lax"`, `https_only` off — **no HTTPS
enforcement** (homelab HTTP; the cookie is single-admin convenience, not a
cloud boundary). Max age is `BOR_SESSION_MAX_AGE` (default `43200` = 12 h).

### Access tokens

The admin can hand out access without sharing the admin password.

1. **Generate:** sign in → **Tokens** in the navbar. Type a label and hit
   **Generate** — a `bor_` + 32-hex token appears in the *shown once* block.
   Copy it now: only its SHA-256 hash is stored.
2. **Use:** the holder enters the token at the in-app gate. The browser
   caches it in `localStorage["bor.token"]` for silent re-auth on reloads.
3. **Revoke:** click **Revoke** on the token's row. Revocation is immediate.

A token user can chat and open documents — nothing else. The RAG catalog,
git sources, tuning, and history stay admin-only.

---

## Using the App

### Updating the documents

The knowledge base is refreshed by **re-running the import**. It is
idempotent and delta-based: each file's RAW BYTES are hashed (sha256,
streamed) and checked against the last index's stamp *before* any
extraction or vision work — an unchanged file costs one hash + one row
lookup (no re-extract, no vision, no re-embed), while a changed file
re-indexes fully. That row lookup reads **metadata only** (phase 185:
`Document.content` is deferred), so a full-resync walk's cost is
metadata-bound instead of corpus-bound; the rare unchanged file whose
`summary` is still NULL re-reads its stored content — one query, one time,
by design — because that backfill genuinely needs the text. The one
exception is a PDF row that still claims a
page strip: it is re-gated from its current bytes (no vision, no LLM —
see "The strip repairs itself" below). A refresh after a normal editing
session takes seconds — pictorial PDFs included (their expensive
first-index vision burns only once per file version):

```bash
# After editing/adding/removing notes:
uv run python -m scripts.import_docs                 # re-index what changed
uv run python -m scripts.import_docs --prune         # also drop deleted files

# Point it at extra directories (repeatable):
uv run python -m scripts.import_docs --source ~/SomeOtherDocs
```

With **git-based sources** each run first pulls the latest commits of your
repos, so this same command is the whole update loop: *commit → re-run*.

Then check the **RAG** page — the knowledge-base catalog at
`http://localhost:8000/sources.html` (the navbar labels it **RAG**; the
separate **Sources** page, `/git-sources.html`, is the source registry):
the *documents* / *chunks* counters and *last indexed* timestamp should
reflect the new files.

- `--prune` removes a pruned file **completely** — its `documents` row and
  its `chunks` rows (the embeddings) go together, the latter through the
  database's own `ON DELETE CASCADE`, so the cost of a mass prune is
  bounded by documents, not chunks (see *A mass prune costs documents,
  not chunks* under [Sync from the UI](#sync-from-the-ui)). The CLI run
  follows the same two phase-184 knobs as the UI sync
  (`BOR_SYNC_SESSION_RESET_FILES`, `BOR_SYNC_PRUNE_BATCH`): it loads only
  the prune's six columns and commits every Nth delete (`0` on either knob
  restores the pre-phase-184 run).
- The import prints one line per file and ends with a greppable summary,
  so it is safe to run from cron or after every commit.
- Indexed formats: **md, markdown, txt, yaml, yml, json, py**, the Podman
  quadlet family (**container, network, volume, image, pod, kube, swap,
  os, endpoint**), **j2** Jinja templates — and, **opt-in**, **pdf**
  (add the `pdf` token to `BOR_IMPORT_EXTENSIONS`; see
  [PDF documents](#pdf-documents-phase-126)).
- Hidden files and common cache dirs (`.venv`, `node_modules`, `.git`,
  `__pycache__`, etc.) are skipped — unless the per-source **Index hidden
  folders** checkbox on the Git sources page is on for that source
  (phase 105): then dot-prefixed folders/files with an importable
  extension are indexed like visible ones, and switching it off prunes
  them on the next sync.
- Non-markdown files get format-aware chunking (YAML keys, JSON keys,
  Python classes/defs).
- With `BOR_IMAGES=1`, standalone images (`png, jpg, jpeg, webp, gif,
  bmp, svg`) are indexed as documents too — `svg` is the one **vector**
  member, indexed from its own markup text with no vision call and
  previewed as a graphic (phase 181) — see
  [Image support](#image-support).
- A run that changed the KB also regenerates the **KB overview** and the
  **folder summaries** (the drill-down catalog's stored per-folder
  descriptions) with the `lite` model — best-effort: a summary failure
  never fails the import. The stored overview ends with a deterministic
  `Sources: <name> (<k> doc|docs), …` line — every source with its exact
  document count, catalog order, computed without a model call — so no
  source (not even a 1-doc one) is invisible to `<knowledge_base>`
  (phase 162); the `LLMError`/zero-doc paths leave the previous row
  (tail included) untouched. Because the sync only rewrites the folders a
  change touches, a one-shot
  `uv run python -m scripts.regen_folder_summaries` refreshes EVERY
  stored folder summary in one run (full mode: every candidate folder
  is regenerated, owner-edited rows are kept, one `lite` call per
  folder, one cron-safe summary line; exit 0, non-zero with a named
  stderr line on a boot/DB failure).

### Git-based sources

Rather than pointing the import at local folders, point it at
**git repositories**. The admin **Git sources page** (`/git-sources.html`)
is the primary management surface — add or remove repositories there and
the list is stored in Postgres.

```env
# .env — the fallback list (fresh setups, or until the admin page
# stores a source; the page becomes the source of truth)
BOR_GIT_SOURCES=https://git.reeseapps.com/reese/homelab.git,git@github.com:reese/deployments.git
BOR_SOURCES_DIR=~/bor-sources   # default; each repo lands in <dir>/<repo-name>/
```

- Public repos need no setup: auth is whatever the machine supplies —
  HTTPS via the OS credential helper, or SSH via your key. The API
  accepts `https://`, `http://`, `ssh://`, `git@`, and `file://` URLs
  (phase 135: a `file://` local-transport repo — e.g. on a mounted
  volume — is a valid source too).
- Private repos: the add/edit form on the Git sources page takes an
  optional **masked token** field (phase 121). The token is stored in a
  dedicated DB column and injected into the clone URL only at
  sync/clone time — it is never returned by any API response or shown
  in the UI (legacy rows that already embed `user:token@` in the stored
  URL keep working — the token is sanitized on output).
- A git source can pin the **branch it mirrors** (phase 135): the
  add/edit form's optional branch field (or the API's `branch`) stores
  the branch in the `git_sources.branch` column — a fresh clone checks
  that branch out, an existing checkout tracks it (`git fetch origin
  <branch>` + `git merge --ff-only FETCH_HEAD`), and a checkout sitting
  on a *different* branch fails the sync loudly with a remove-and-
  re-add message (never a silent switch). Rows without a branch —
  every pre-existing row and the env-fallback rows — track the
  checkout's default branch (byte-identical clone behavior).
- Every run **clones** (first time, full history) or brings an existing
  checkout to the **head of the tracked branch** (phase 136: `git fetch
  origin <branch>` + `git merge --ff-only FETCH_HEAD`) and then indexes
  the checkout. If the remote was **force-pushed** — a fast-forward is
  impossible — the checkout is reset hard to that head with one loud
  log line **instead of failing the sync**: the checkout is a cache of
  the remote head, and the KB is rebuilt from it on the same run, so a
  force push is a logged, self-healing event. A genuinely broken fetch
  (dead network, a branch deleted upstream) still fails the sync
  loudly. A legacy shallow checkout is unshallowed once on its first
  post-phase-107 sync (`git fetch --unshallow` — the one-time
  self-heal) so commit dates stay real.
- A failed sync aborts the run — no partial junk.

### Archive upload sources

Upload a `.tar`, `.tar.gz`, `.tgz`, or `.zip` archive to make it a source.
The form on the Git sources page — and the `POST /api/git-sources/upload`
route behind it — accepts archives and unpacks + registers them (no scan —
the upload indexes nothing; the **Sync sources** button on the **RAG**
page (`/sources.html`) performs the scan, which leaves room to edit the
new source's ignore list first).

- Re-uploading the same filename replaces the source **in place** (no
  second folder, no duplicate row).
- After the 202 the form settles into the ready-for-sync line
  "Uploaded <source> (<stamp>) — press Sync sources to import it." —
  the completion stamp (phase 154: the same date-qualified freshness
  rule as the sync badge — `HH:MM` on the same local day,
  `M/D, h:MM AM/PM` otherwise, `M/D/YYYY` across years) rides the
  line on EVERY render path, so a line that outlives its upload is
  self-dating: after a reload the source name is gone (it was
  page-local) but the stamp survives, coming from the server
  payload's `finished_at`.
- Zip-bomb guard: absolute member paths, `..` traversal, and symlinks
  escaping the unpack folder are rejected.
- Archives unpack under `BOR_UPLOAD_DIR` (default `~/bor-sources/uploads`).
- `BOR_UPLOAD_MAX_MB` (default 512) caps both compressed and extracted size.

### Local directory sources

A plain directory can be a first-class source too. It shares the git
sources' **one table** (the `git_sources` registry with a `kind`
discriminator: `git` | `local`), **one admin page**, and **one Sync button**.

- Register via API: `POST /api/git-sources` with `{"kind": "local", "path": …}`.
- Sync walks it directly — no clone, no checkout copy.
- The directory is re-verified to exist at sync time; a missing directory
  fails the run loudly.

### Sync from the UI

The **Sync sources** button on the **RAG** page (`/sources.html`, admin
only) runs the whole source refresh in one click:

1. **clone/pull + walk** every configured source — git repos and local
   directories, including uploaded archives.
2. **re-import with prune** — files deleted upstream leave the index.
3. **regenerate the KB overview** (the `<knowledge_base>` outline every
   chat turn injects) — only when the import changed the knowledge base.
4. **refresh the folder summaries** (the drill-down catalog's stored
   per-folder descriptions) — a changed knowledge base rewrites ONLY the
   folders the changed files touch (each file's folder chain up to its
   source root — phase 127: one changed file re-stamps a handful of
   rows, not the whole table); an unchanged re-sync instead gap-fills
   any missing rows (`only_missing`).

- States: while polling `GET /api/sync/status` every 2 s, the button
  becomes the spinner (icon only — its label parts are hidden while a
  run is in flight, and its accessible name is the full live label),
  and the full live detail — "Syncing… <file> (n/m)" during the import
  phase (the live file, its `(n/m)` file position, and — while a file's
  embed batch is in flight — the pinned `· chunk n/m` embed position,
  batch granularity: the first chunk of the in-flight batch over the
  file's total content chunks; and — while a PDF's images are being
  described — the `· image n/m` vision position: the 1-based image
  about to be described over the file's total images, ONE continuous
  counter across the PDF's three vision spans, so a long image pass
  visibly climbs instead of sitting frozen on the last chunk
  position) — shows in a text box under the button row,
  wrapping (never clipped); the same untruncated label rides the
  button title and the box, which is the SOLE `aria-live` region for
  the sync area (`role="status"`, polite — phase 150 folded the
  pre-phase result-line announcer into the box). At settle the box
  STAYS visible to carry the result line — the counts line
  ("N added · …") after a success, "Sync cancelled" after a cancel, each
  trailed by the run's peak memory (" · peak 1936 MB", phase 187; nothing
  at all when the reading is null) —
  and the button leaves the running state (the "Synced <stamp>" pill
  after a success — the date-qualified freshness stamp (phase 154):
  `HH:MM` on the same local day, `M/D, h:MM AM/PM` otherwise
  (`M/D/YYYY` across years — a badge that tells the owner *when*),
  the bare "Synced" when the stamp is missing — the retry-ready
  "Sync sources" pill after a failure); the box hides only when its
  text is empty (a failed sync
  — the banner + modal carry the error — and the true idle). No
  client-side timeout — a clone + embed can legitimately take minutes.
- One sync at a time: a second trigger while a run is in flight gets a
  `409`.
- **A long clone never freezes the server** (phase 179). The run's git
  steps (`clone_or_pull` / `file_commit_dates`, the blocking
  `subprocess` calls) execute in a worker thread
  (`asyncio.to_thread`), so the event loop keeps serving the whole
  HTTP surface for their duration — every page, `GET /api/health`, and
  above all the 2 s `GET /api/sync/status` poll that watches the run.
  Before this a big first clone stalled the loop thread and the entire
  app looked dead until git returned.
- **A mass prune costs documents, not chunks** (phase 183). A pruned file
  leaves the index completely — its `documents` row AND its `chunks` rows
  (every embedding) — but the chunk rows go **in the database**, through
  `chunks_document_id_fkey`'s `ON DELETE CASCADE` (alembic 0001), because
  `Document.chunks` is declared `passive_deletes=True`. So a prune of 3,000
  documents issues batched `DELETE FROM documents` statements and **zero**
  `SELECT`s of the 768-dim vectors it is about to remove: the memory a
  prune costs is bounded by the *document* count, not by chunks ×
  ~24.8 KiB of hydrated embedding. Before this, `session.delete(doc)`
  loaded every child collection first, and the 2026-10-15 sync's
  3,008-document prune materialised ~72k embeddings (~720 MB) on a 3,904
  MB VM with no swap. The same holds for removing a source on the
  **Git sources** page (the other mass-delete surface). Pinned by
  `tests/integration/test_prune_cascade_statements.py` — the statement
  count, the zero-orphan-chunk-rows contract, and a live-FK drift guard
  that fails the suite if the cascade clause ever disappears; the owner
  flow itself (delete upstream → re-sync → exactly those documents and
  chunks gone) is pinned by `tests/e2e/test_prune_cascade_cleanup.py`.
- **A sync's memory is bounded per RUN, not per knowledge base** (phase
  184). One import run holds one session, and `expire_on_commit=False`
  (`app/db.py`) means a commit releases nothing — so before this phase the
  run kept every row it touched (each unchanged document with its full
  ~15 KB `content`) until it finished, and the prune read every document of
  every pruned source whole, deletes held in ONE transaction. Now the walk
  releases its session every `BOR_SYNC_SESSION_RESET_FILES` files (plus once
  per source), and the prune (`_prune`) reads only the SIX columns its
  guards and its copy cleanup need (`id`, `source`, `path`, `is_image`,
  `image_path`, `page_count` — never `content`) ordered by id, committing
  every `BOR_SYNC_PRUNE_BATCH` deletes. **Where the operator sets the
  ceiling:** those two `BOR_`-prefixed settings (defaults `200` /
  `500`, live on the next sync without a restart, `0` on either one restores
  the exact pre-phase-184 run as the no-redeploy rollback). Counts, log
  lines, `changed_paths`, the phase-122/126 prune guards and the database
  are byte-identical either way — pinned by
  `tests/integration/test_import_session_retention.py` (the identity-map
  bound, the no-`content` prune `SELECT` with no per-row lazy load, the
  3-commits-vs-1 cadence, both prune guards on light rows) and end to end
  through the RAG page (`/sources.html`) by
  `tests/e2e/test_sync_session_reset.py` — two
  module apps on their own ports (`E2E_APP_PORT_SYNCRESET` /
  `E2E_APP_PORT_SYNCRESET_ZERO`), one per cadence: the ten-file walk lands
  every document exactly once with the totals the stat cards show, the
  re-sync is all-`unchanged` with an unchanged chunk total, a five-document
  prune at batch `2` clears both folders with zero orphan chunk rows, and a
  `0`-cadence run of the same fixture is byte-identical to the `3`-cadence
  one (per-path chunk counts, `content`, `summary`, both hashes, the counts
  line).
- **A full walk reads row metadata, not the corpus** (phase 185). The four
  row lookups the unchanged fast path shares (`_index_file` and the PDF /
  PDF-embedded-image / standalone-image branches) go through ONE statement,
  `_doc_row_stmt`, which defers the single column the fast path never
  reads — `Document.content`. The 2026-10-15 sync inspected ~27k files and
  re-indexed 217, yet hydrated all 17,407 rows whole (avg 15,217 chars =
  **252.6 MB of markdown**, held for the whole run by `expire_on_commit=False`
  and phase 184's stamp); now an inspected row carries file_hash, summary,
  dates, page count — and no markdown. Exactly ONE column is deferred, and
  there is deliberately **no `raiseload`/`noload`**: the phase-118
  summary-NULL backfill genuinely reads the stored content, so that one row
  pays one extra `SELECT documents.content … WHERE id = …` (per-row,
  one-time, already rare) instead of crashing. Counts, log words, the D1
  manual-date lock, the phase-163 strip re-gate and the backfill result are
  byte-identical — pinned by
  `tests/integration/test_import_fast_path_columns.py` (the captured
  fast-path `SELECT` names no `documents.content`, one `SELECT` per
  unchanged file with a summary, exactly one lazy follow-up for the
  backfill), `tests/unit/test_importer_defer_content.py` (`content` is
  the only deferred column, all four loads share the helper, no
  `raiseload`/`noload` anywhere in the importer) and, through the UI,
  `tests/e2e/test_fast_path_deferred_content.py` (a hand-nulled summary
  backfills to its byte-identical text from the RAG-page
  (`/sources.html`) re-sync with
  exactly one `is_summary` chunk, an unchanged doc's viewer still shows
  the full stored text, and a rewritten file re-indexes to the new text
  with the next sync all-`unchanged` again).
- **The commit-date walk streams** (phase 186). A git source's per-file
  last-commit dates come from one `git log --name-only --format=@@%cI`
  over the checkout — a walk that scales with the repository's **entire
  history**, not its working tree (the incident box measured 346,829
  lines / 15 MB for one source, and the removed `server`/nextcloud
  source showed how little that has to do with the files on disk). It no
  longer arrives as one string: `stream_git_lines` (the second,
  documented spawn site beside `run_git`) yields it line by line off the
  pipe, so no `git log` is ever held in RAM, and stderr is drained on a
  worker thread so a chatty git cannot deadlock the pipe. The parser
  takes any iterable of lines (`_parse_commit_dates(str | Iterable[str])`
  — a string still behaves exactly as before), so the grammar is
  untouched. The dates are byte-identical — first-sighting-wins, the same
  fail-soft arms (`{}` + the same warning lines + the mtime fallback),
  the D1 manual date untouched — and what IS still retained is the
  per-source date **dict**, held by the run for its whole duration
  (phase 186 A3: it is ~10 % of the cost this removed, and bounding it
  would change dates, not just memory). Pinned by
  `tests/unit/test_git_sync.py` (the stream, the `run_git`-shaped
  `GitSyncError`, the 340 KB-stderr no-deadlock case, the
  terminate-on-abandon case, and `test_only_two_git_spawn_sites`, the AST
  pin that enumerates EVERY `subprocess.*` spawn in `app/` and
  `scripts/` — the two git sites plus the one non-git exception, the
  boot-time Valkey probe in `app/core/caching.py` — so a new spawn
  anywhere fails the suite), `tests/integration/test_git_file_dates.py`
  (the streamed dict EQUALS the buffered one across a commit-1-only
  file, a commit-2-modified file, a subdirectory file and a
  never-committed file that falls back to its mtime) and, through the
  browser, `tests/e2e/test_git_log_stream_dates.py` (a real two-commit
  `file://` source synced from the RAG page (`/sources.html`) shows each
  file's own last-commit date in the table and the viewer, an
  uncommitted file its mtime, the owner's manual date survives the
  re-sync, and an unchanged re-sync moves no date at all). One
  pre-existing quirk is recorded while we are here, **unchanged** and
  pinned by that suite: with git's default `core.quotepath=true` a
  non-ASCII path reaches the walk quoted and octal-escaped
  (`"notes/caf\303\251.md"`), the parser's whitespace split and
  `\`→`/` normalisation turn it into a key that can never match the
  importer's path, and such a document therefore shows its **file
  mtime** — the day the checkout last wrote the file — instead of its
  commit date. Fixing it means changing the quoting grammar — a date
  change for every non-ASCII document already in a KB, so it is a future
  phase's call, not a transport phase's.
- **The run's memory is a logged number and a line on the page**
  (phase 187). Every sync lifecycle line ends with `rss_mb=` — and every
  terminal with `rss_mb= rss_peak_mb=` — read through
  `app/core/memory.py` (PROCESS RSS, never the host's cgroup figure,
  which counts page cache; an unknowable reading prints `?`, never
  `None`), and the SAME pair rides `GET /api/sync/status` as the
  additive `rss_mb` / `rss_peak_mb` keys (`int | null`). The RAG page
  (`/sources.html`) appends it to the result line it already spoke —
  `3 added · 1 unchanged · peak 1936 MB`, `Sync cancelled · peak 1936 MB`
  — no new control, and nothing at all when the pair is null. "How much
  RAM did that sync use?" is therefore one grep of the app's own log
  **or** one
  glance at the page that started the run, instead of a dig through
  `journalctl` unit-teardown lines from three boots (the 2026-10-15
  incident). The fields are appended, so every older grep still matches,
  and the sampler keeps the pair honest (`peak >= current` — a kernel
  high-water refreshed lazily is raised to the current reading, because a
  peak below the run's own footprint is a false number, not a smaller one).
  Pinned by `tests/unit/test_core_memory.py` (every helper branch, the
  never-raise contract), `tests/unit/test_sync_button.py` (the sampling
  points, the 5 s throttle, the pair's invariant),
  `tests/integration/test_sync_api.py` (the four lifecycle lines, the
  additive keys, the terminals keeping the final pair, the anonymous 403)
  and end to end in the browser by
  `tests/e2e/test_sync_peak_memory_visible.py`. See *Operational notes:
  the sync's memory (phase 187)*.
- **Cancel**: while a run is in flight, a **Cancel** button appears
  next to the sync button; clicking it posts `POST /api/sync/cancel`
  (admin only) and registers a cooperative cancel (`202`) — the run
  stops at the next pipeline boundary (a git clone/pull already in
  flight runs to completion in its worker thread — the request is not
  held, and the rest of the app keeps answering meanwhile) and settles
  into a neutral `cancelled`
  terminal state (`error: null` — intent, not failure: a "Sync
  cancelled" result line, no error banner or modal, the button
  retry-ready) that the status endpoint reports; with no run in
  flight the request gets a `409` (the UI treats it as a harmless
  race — the poll settles). A sync started after a cancel runs clean
  (the importer's content-hash delta re-imports what the cancel
  skipped).

---

## Thinking

The self-hosted `turbo` model reasons before it answers. That reasoning is
streamed with the turn as `thinking` SSE events and shown as **"Thinking"
blocks** — one block per contiguous thinking run, in arrival order
(phase 158) — all wrapped in **one collapsible "Reasoning" bubble** above
the answer bubble (phase 161): the turn renders as a chronological timeline
(Thinking → Tool call (1) → Thinking → … → answer), condensed into a
single bounded live window (320 px, one scroll surface) that follows
the tail while you watch. The Reasoning bubble is the ONE disclosure
(owner refinement, 2026-10-06): the per-run blocks inside it are plain and
never fold on their own.

The bubble is created on the turn's first thinking (or tool) frame and
stays **open** from creation until the turn settles — including a
mid-round reply (the model answered, called a tool, then thinks again:
the reply streams into the answer while the bubble stays open, and the
new thinking run gets its own block inside it instead of resuming on
top of it). It folds when the turn ends, is stopped, or fails. While
open, its summary carries a live status ("Thinking…" /
"Calling <name>…") and reads "Reasoning — N steps" once the turn
settles; a turn with no thinking and no tools renders no bubble at all
(the answer layout is byte-identical to pre-phase).

The per-run blocks are plain and always visible inside the open
bubble: each block appears and fills in live while its run streams,
and the runs simply stop accumulating frames when the first answer
token lands, when the turn ends, or when the turn is
stopped/failed. They are distinct from each other by label + rail:
the brand-ink "Thinking" label on a brand rail, the accent-ink
"Tool call(s) (N)" label on a neutral rail.

The tool calls ride the same timeline: each contiguous run of tool
calls gets its own "Tool call(s) (N)" block (one per run, in
arrival order; always visible while the bubble is open), and the
per-line elapsed-seconds suffix targets the newest line of the
active run.

The timeline is persisted with the saved turn (an optional `segments`
array — the per-run order; the concatenated `thinking` and the `tools`
record stay for history and older clients), so a reload and the shared
page re-render the same timeline **collapsed inside the Reasoning
bubble**, with the step-count summary.

To hide it, set `BOR_STREAM_THINKING=0`.

---

## Agent document tools (ls + read + grep)

Retrieval only puts the top documents in context. When an answer depends on
a file a note *references* ("the exact JSON shape is in
`example-record-file.json`"), the model can extend its own context with
server-side tools — on **grounded** (high-relevance) turns only:

| Tool | Description |
|------|-------------|
| **`ls`** | Lists the knowledge base as a **drill-down tree**, one level at a time: no `path` = every source (document count + stored description), a source name = that source's top level, a `source/folder` path = one level deeper; file lines are `source: X \| path: Y \| title: Z` |
| **`read(path, offset?, limit?)`** | Appends one indexed document's text to context by its combined `source/path`; optional `offset` (1-based first line) + `limit` (line count) read just that part of a large document; very large reads are truncated at `BOR_READ_MAX_CHARS` with a notice that teaches the workflow — grep reports line numbers, then read that part with `offset` + `limit` |
| **`grep(pattern, path?)`** | Searches indexed documents for an exact string (case- and accent-insensitive fixed substring — accents fold: `besta` matches `BESTÅ`, phase 162); returns up to 20 `source/path:line: text` matches; an optional `path` limits to one document |
| **`show_image(path)`** | *(only when `BOR_ANSWER_IMAGES=1`)* shows one standalone image document as part of the answer — image documents only, at most one per answer |
| **`show_text(path, offset?, limit?)`** | *(only when `BOR_ANSWER_TEXT=1`, the default)* displays the named range of a text document to the user **verbatim** — the server shows the bytes and keeps them byte-identical; the model describes them, never retypes them; image documents are refused (use `show_image`); per-answer display budgets enforced server-side |

(See [Answer images](#answer-images-phase-124) for the `show_image` tool and
[Verbatim text display](#verbatim-text-display-phase-182) for `show_text`.)

**The prompt teaches the search discipline (phase 162).** A summary or an
outline is a hint, never an inventory: the system prompt tells the model to run
`ls` with no `path` — the full source list — before ever claiming the knowledge
base lacks something, and to `grep` the artifact's part number or a filename
token. One grep pattern is one string, so a name's split tokens (`frame`,
`white`) and its part number (`AA-1272068`) each need their own grep; and a
note saying a file is gitignored describes its **repo** status, not whether it
is indexed as a source.

**Git-source provenance (phase 145).** When a document comes from a git
source, its `read` result also carries an `origin:` line (the repository's
URL, credentials stripped) and, when the source pins one, a `branch:` line.
The prompt teaches the model to include that original source in its answer —
a link to the repository with the document's path named (and the pinned
branch when one is shown) — so you can open the complete original in one
step rather than asking again for the source. Local (non-git) sources carry
no `origin:` line, so nothing is added and no link is invented. The same
origin rides the document-content payload (`origin_url` + `origin_branch`
when pinned), and the document viewer — the chat's doc modal AND the full
`/document.html` page, one render site — shows it as an **Original**
external link in the document's meta row (the full repository URL as the
link text, opening in a new tab; the pinned branch rides the link's
title). Non-git documents show nothing new.

Each call is executed against Postgres only (no extra LLM round trip) and
streamed as an SSE `tool` frame. In the chat, each call shows a transient
**calling-tool status** alongside "thinking" (the send button keeps its
busy state — "Stop" — for the whole turn).

**The search line shows where it searches (phase 146).** When a `grep`
re-searches one document (the optional `path`), the chat's tool line and
the calling-tool status show the document the search runs in —
`🔎 Searching for <pattern> in <source/path>` — so a whole-KB search and
a re-search of a single document read as distinct searches instead of
looking like the model is repeating itself. The scope rides the SSE
`tool` frame additively (`path`, present on scoped greps only) and is
persisted on the saved turn's tool record, so it re-renders after a
save → restore; a bare `grep` renders exactly as before (frame, line,
status, and saved shape all byte-identical to pre-phase — old saved
turns restore unchanged).

The model may call tools as many times as needed, bounded by a round cap:

| Env | Default | Meaning |
|-----|---------|---------|
| `BOR_AGENT_MAX_ROUNDS` | `10` | hard cap on agent tool rounds per grounded turn (0 = no tools, the kill switch); every tool result carries the model's remaining count (phase 160) |

At the cap, further tool calls are **rejected** (never executed, never
shown) and the turn ends with a final no-tools reply: the model is
coached to answer from the results it already has and to tell the user
they can say "continue" to keep researching; a reply with no visible
content answers with a deterministic "budget spent" fallback line
instead of an error frame (phase 160, the recorded revision of the
phase-71 cap-path terminal). The `done` frame carries an additive
`budget_exhausted` key (ABSENT on every non-cap turn) that drives the
**Continue researching** button on the answer's meta row — see the
[Chat (`/`)](#chat-) section.

---

## Tuning your answers

*Admin-only* — sign in first. If an answer isn't quite right — too chatty,
wrong assumption, missing context — **tune** Brain right there:

1. Press **"Tune"** in the meta row under any completed answer (deflected
   ones included).
2. Type a short instruction (1–2000 chars), e.g. *"be more concise"* or
   *"assume I'm on NixOS"*, and **Save**.

The note is stored in Postgres (`steering_notes`) and read into the
**system prompt of every subsequent chat turn** as a `<tuning>` section
(numbered, oldest first, capped at `BOR_STEERING_MAX_CHARS` chars — default
8000, overflow marked `[…truncated…]`).

List or remove notes from the **"Tuning"** button in the chat header (count
badge, newest-first, per-note delete). The API is stateless JSON if you
prefer curl:

```bash
curl -s localhost:8000/api/steering                                  # list
curl -s -X POST localhost:8000/api/steering \
     -H 'Content-Type: application/json' -d '{"note": "be more concise"}'
curl -s -X DELETE localhost:8000/api/steering/<note-id>              # remove
```

---

## How retrieval works (hybrid)

Every question is embedded and also lexically tokenized (OR-joined, English
stemming) and searched **twice** against Postgres:

1. **Vector** — pgvector cosine top-N (default `BOR_HYBRID_VECTOR_CANDIDATES=100`) over
   the `ix_chunks_embedding_hnsw` HNSW index (phase 188), searched at
   `BOR_RETRIEVAL_EF_SEARCH` (default **`400`**, set per transaction with
   `SET LOCAL` — never a global setting). 400 is not a guess: the live battery
   in [`benchmarks/phase188_live_gate/`](benchmarks/phase188_live_gate/README.md)
   measured the graph recovering the exact top-100 for only 0.9814 of rows on
   average (0.94 worst) at `100` — the candidate count — against 1.0000 at 400,
   with every A8 verdict, top-5 set and `top_score` unchanged at both depths
   (the honesty thresholds were never touched). A value below the candidate
   count is **clamped up** to it with one WARNING line in the server log
   (`retrieval: ef_search=… raised to …`): a smaller `ef_search` returns
   fewer candidates than the tier promises, which is a silent recall
   regression the honesty gate can turn into a deflect. Raise it (max 1000)
   if recall matters more than the tier's latency.
2. **Lexical** — a stored `tsvector` (GIN-indexed) matched with `to_tsquery`,
   top-N by `ts_rank` (default `BOR_HYBRID_LEXICAL_CANDIDATES=30`), led by
   **name hits** — documents whose *path* matches a question token (`gitea`
   → `gitea/…`, `qwen38` → `qwen3.8-27b-…`; phase 119). Name normalization
   is **accent-folded** (phase 162): a question's `BESTÅ` folds to `besta`
   and matches the ASCII filename, and an accented filename (`BÅsta.md`)
   matches an ASCII question — both directions, stdlib fold only.

The two ranked lists are fused with **Reciprocal Rank Fusion**
(`score = Σ 1/(k + rank)`, `BOR_RRF_K=60`) — a chunk in both lists scores
nearly double, which lets a name-your-tool question ("gitlab") find its own
document even when the question embeds close to generic templates.

The **honesty gate** (A8) then answers (HIGH) when the best cosine is ≥
`BOR_RELEVANCE_THRESHOLD` (default `0.62`) **or** when at least one chunk
matched lexically (`fts_hits > 0`) **and** the best cosine clears
`BOR_LEXICAL_SUPPORT_FLOOR` (default `0.35`) — the vector signal must
corroborate a lexical hit (A8 revised 2026-09-14). It deflects (LOW)
otherwise, including a weak single-token hit with vector-unsupported
docs. The top `BOR_TOP_N_DOCS` full documents are what the LLM sees.

---

## Document summaries

Raw yaml/json/py/txt embeds badly — flags and keys are not language, so
retrieval can miss exactly the documents that are all configuration. At
import time, **every** new or changed document gets a `lite`-model summary —
**markdown included**: the summary is a natural-language retrieval target
and the seed text for the onboarding suggestion blocks (phase 118 retired
the old "markdown is already natural language" exclusion):

* The summary is stored on `documents.summary` **and** indexed as one extra
  embedded chunk (`chunks.is_summary`, position −1), so hybrid search has a
  natural-language target to hit instead of the raw text.
* The last line is a **code-deterministic** pointer — `Source: <source>/<path>`
  — appended by the app, never model-generated.
* The model only sees the first `BOR_SUMMARY_MAX_CHARS` (default 12000)
  characters; overflow is cut and marked with `[…truncated…]`.

Summary generation is best-effort: if `lite` fails, the document is still
indexed (without a summary), the failure is logged, and counted in the
import summary line (`summaries=N summary_errors=N`).

---

## Image support

One toggle gates both image capabilities: `BOR_IMAGES` (default
**off**). Enable it **only when your chat model
(`BOR_LLM_CHAT_MODEL`) supports vision** — the chat model is the vision
model: it describes image documents and answers questions with attached
images. The embedding model never sees pixels in either flow.

### Image documents

With `BOR_IMAGES=1`, a standalone image file — inside an uploaded
source archive, or as a file in a git/local source walk — becomes a
first-class document:

- **Described by the chat model.** On first index, a single multimodal
  chat-model call describes the image. The description becomes the
  document's `content` **and** `summary` — and is the *only* text that
  gets embedded, so hybrid search has a natural-language target to hit
  instead of pixels.
- **The bytes persist.** Each ingested image is copied to
  `BOR_IMAGE_DIR/<uuid>.<ext>` and served from that copy (uploads are
  replaced and checkouts re-cloned, so the served image outlives its
  source file).
- **Retrieved & displayed.** Image docs appear in the RAG catalog
  with a small lazy-loaded thumbnail (progressive enhancement — a fetch
  failure falls back to the doc icon), the document viewer renders the
  image with the description below it as **wrapping prose** (phase 165 —
  the full description reads by scrolling down, never sideways), and a
  cited image doc renders
  **in the chat's sources images row** (phase 130: its own
  `.msg-meta.msg-images` cluster below the chips row — no chip for an
  image doc, the figure is the citation; alt text = the description;
  a failed image load replaces the figure in place with the plain chip,
  so the citation survives a lost byte copy). The agent's `read` tool
  returns the description with a marker telling the model it is reading
  an image document.
- **Prune guard.** While `BOR_IMAGES` is **off**, syncs never prune
  existing image documents — an image doc is invisible to the walk, not
  a deleted file. With the toggle on, a deleted image file prunes its
  doc and its copy.
- **Fail-soft.** A failed or empty description skips that image (no
  row, no copy), logs a warning, and counts in the import summary line
  (`images_failed=N`) — the sync continues.

Accepted formats are `png, jpg, jpeg, webp, gif, bmp, svg`
(`BOR_IMAGE_EXTENSIONS`, case-insensitive) — a separate set from
`BOR_IMPORT_EXTENSIONS`; images are never added via that variable, the
toggle is the single knob. With `BOR_IMAGES=0` an `.svg` is therefore a
plain **text** document — raw XML in the viewer, no image copy, no
thumbnail — and the walks never admit it as an image (no special-casing
of the off state).

**SVG is the one vector member (phase 181).** An `.svg` takes the same
image path — its bytes are copied to `BOR_IMAGE_DIR` and served as
`image/svg+xml`, so the chat figure, the document modal and the
`/document.html` viewer show the *picture* instead of the XML — but it
is **never sent to the vision model** (vision endpoints reject
`image/svg+xml`). Its searchable text comes out of the file itself: the
`<title>`, `<desc>` and `<text>` words, or — for a drawing that carries
no words at all — one honest structural line built from the file (its
name, its shape census, its `viewBox`). A file the extractor refuses
(over 2 MiB, carrying a `<!DOCTYPE`/`<!ENTITY` declaration, or malformed
XML) fails soft like any other image: no row, no copy, `images_failed`.
A successful one logs one `import: svg indexed source=… path=…
structural=… chars=…` line and adds to the log-only `svg_indexed=N` on
the import summary line (`added`/`updated` count it like any other image
doc). The next sync also **converts** an `.svg` that an earlier run
indexed as plain text (its bytes hash is unchanged, so a hash-only fast
path would freeze it as XML forever) — existing libraries need no
re-import by hand. In the viewer the graphic **spans the image card**: a
`viewBox`-only SVG
carries no intrinsic size, so `doc.format === "svg"` adds
`.doc-image-img--vector` (`width: 100%` + `object-fit: contain`) instead
of letting the drawing sit in the browser's 300×150 default box; the
chat figure and the RAG-table thumbnail keep their fixed caps (96px /
48px) and the modal is the full-size preview. Question images (the
paperclip) stay raster-only: an `.svg` upload is a 422.

Those bytes are served **sandboxed**: because the
`/api/documents/<id>/image` URL is directly navigable (not just an
`<img>` source, which can never run SVG script), an SVG response carries
`Content-Security-Policy: default-src 'none'; sandbox` so a stored
drawing can never execute script or act with the app's origin — raster
image responses keep their headers exactly as they were.

### Question images

With the toggle on, the composer gains a **paperclip** button. Attach
one image to a question and the chat model answers with text **and**
image:

- **Server-stored, path-persisted.** On send the file is first uploaded
  to `POST /api/chat-images` (capped at `BOR_CHAT_IMAGE_MAX_MB`,
  default 10 MiB — over is a 413) and stored under `BOR_CHAT_IMAGE_DIR`.
  Saved and shared chats carry the returned *path* — never base64.
- **Raster only (phase 181).** The accepted upload extensions are the
  **raster** members of `BOR_IMAGE_EXTENSIONS` — an `.svg` is a 422. The
  vector family belongs to indexing, not to question attachments: a
  vision endpoint rejects `image/svg+xml`, and the question-image serve
  route deliberately never hands out a `.svg`, so accepting one would
  store an unanswerable, unservable file.
- **Visible while uploading.** While the upload is in flight the
  preview strip shows a spinner + *"Uploading image…"* (its remove ✕
  disabled) and the Send button is disabled with the *"Uploading…"*
  label — a second press is dead, visually and functionally (no second
  upload, no second turn). On success the state clears and the turn
  proceeds; on failure it clears, the banner shows, and the typed text
  + the attachment stay for the retry (phase 132).
- **Current turn only.** The model receives a multimodal message for
  this question; prior turns' images are never replayed into history
  (the text of a prior turn stands alone).
- **Rendered everywhere.** The image renders in the user bubble, after
  a page reload, and on the shared chat page; a failed load degrades to
  a small "image unavailable" note (never a broken-image icon).
- **Blocked on failure.** If the image upload fails, the send is
  blocked with a banner — the question is never sent without the image
  the user attached.
- **Toggle off** ⇒ the attach control is absent, and an API request
  carrying an image gets an error frame with the hint: *Enable
  `BOR_IMAGES` in the server's `.env` (and restart) to ask with an
  image.*

A question image is **never indexed as a document** — it is turn-local
storage, not a source.

### Answer images (phase 124)

A second, independent toggle — `BOR_ANSWER_IMAGES` (default **off**): when
on, the grounded chat model may show a standalone image document as part
of its answer:

- **The model decides.** The agent is offered a fourth tool,
  `show_image(path)`, and calls it only when the image itself contains
  the answer (a diagram, screenshot, or photo of the thing asked about) —
  never for mere topical relevance. At most **one** image per answer
  (enforced server-side); the text answer is always given in full — the
  image supplements, never replaces.
- **Rendered as an answer figure.** A shown image appears as a
  first-class figure below the answer text and above the source
  clusters (the chips row, then the images row — phase 130) — on the
  live turn, after a page reload, and on the shared chat page; a failed
  load collapses the figure (never a broken-image icon).
- **Image citations on zero-read turns.** A retrieved image document in
  the suggested tier is auto-cited: its figure (in the images row — no
  chip, phase 130) appears even when the model never `read`s it — the
  model's seed block already carries the image's full description, so a
  real model has no reason to read it (this applies with `BOR_IMAGES=1`,
  any `BOR_ANSWER_IMAGES`).
- **Off = byte-identical.** No fourth tool offered, no figure, no
  `answer_image` key in any SSE frame or saved record.

---

## Verbatim text display (phase 182)

The mirror of the answer-image trust model, for words: when the user asks to
see part of a document — a stanza, a passage, an exact range of lines — the
model **names the location and the server shows the bytes, byte-for-byte**.
The model never retypes quoted text, so a displayed quote cannot drift.

- **The model decides.** The grounded agent is offered a display tool,
  `show_text(path, offset?, limit?)`, and calls it when the user wants to
  SEE, COPY or RUN a file — a quadlet, a config, a script, a unit file —
  or read its exact wording. Optional
  `offset` (1-based first line) and `limit` (line count); an overshooting
  `limit` is clamped, an `offset` past the last line is refused with the
  true line count.
- **What the prompt teaches.** With the switch on, the grounded `<tools>`
  section gains one clause (after the `show_image` clause when that rides
  too) saying the two things the tool description cannot: the user copies
  from **what is displayed**, not from the model's prose — so after a
  `show_text` call the model describes the lines instead of retyping them
  — and the inline locator fence for placing a displayed block mid-answer:
  a line of three backticks + `bor` + the combined `source/path` + an
  optional `12-40` range, with an **empty body** the app replaces with the
  stored text. Line numbers come from `grep`/`read`, never from a guess —
  when unsure, the range is omitted and the whole document is displayed.
  `BOR_ANSWER_TEXT=0` removes the clause too, so the prompt never teaches
  a tool the model was not given.
- **Server-side budgets.** At most `BOR_SHOW_TEXT_MAX_BLOCKS` blocks per
  answer (a second identical show is refused — it is already on screen);
  each block is capped at `BOR_SHOW_TEXT_MAX_CHARS`, cut at a line boundary
  with a truncation marker. Image documents are refused with a pointer to
  `show_image` — the two display tools never convert into each other.
- **The bytes ride the `done` frame.** A turn that displayed text ends
  with one extra, additive SSE key: `done.answer_text`, the list of blocks
  in **display order** — answer order first (one per ` ```bor ` fence, the
  fence controls placement), then the `show_text` blocks the answer never
  located. Each entry is
  `{source, path, title, start_line, end_line, total_lines, text, truncated, locator}`
  where `text` is the **server's** slice of `documents.content` (never
  model output) and `start_line`/`end_line` are the server's resolved
  1-based inclusive extent — the numbers a caption shows can never be the
  model's claim. `locator` is the canonical matching key
  (`source/path`, or `source/path#12-40` for a range). There is no new
  bytes route and no eighth SSE event type: the bytes travel inline, and a
  saved or shared chat renders its panels from its own record. The same
  array persists on the saved brain record (`bor.chat.v1` `answer_text`),
  so a reload and an anonymous shared page show the same blocks.
- **Deflected turns resolve nothing.** A ` ```bor ` fence written into a
  deflected answer leaks no KB text — the disclaimer turn resolves no
  block and its frame carries no `answer_text` key.
- **Off = pre-phase.** `BOR_ANSWER_TEXT=0` removes the tool from every
  offer **and** its `<tools>` clause (the grounded system prompt is
  byte-identical to pre-phase); answers return to composed text (the model
  quoting from what it read). A turn with no blocks serializes **without**
  the key — absent, never `null`, never `[]` — so every block-free frame
  and every pre-phase saved record keeps its exact pre-phase bytes. The
  SERVER surface is byte-pre-phase; the CLIENT rule is flag-independent, so
  a stray ` ```bor ` fence in a flag-off answer still cannot paint the
  model's copy — with no `answer_text` to hydrate it, it settles to the same
  honest sentence (the phase E2E's App B spec).

- **The client never paints the model's copy.** The one shared markdown
  renderer (`frontend/assets/markdown.js`) takes a SECOND, optional
  argument — `renderMarkdown(md, { verbatim: true })` — and with it ON a
  ` ```bor ` locator fence is replaced, info line, body and closing fence
  included, by a pending panel:
  `<div class="verbatim verbatim-pending" data-bor-locator="…">` holding the
  `source/path` caption and the live-region line *Retrieving the exact
  text…*. The locator is normalized client-side by the exact mirror of the
  server's `canonical_locator` (`source/path`, or `source/path#12-40` —
  digits only, so the two sides match on a string compare), the model's
  bytes appear NOWHERE in the output, and a fence the model has not finished
  typing shows that same pending state instead of raw backticks. An
  unparseable locator degrades to an ordinary code block with the model's own
  body — no placeholder, no badge. With the argument OFF — every other
  surface in the app: the document viewer, the thinking blocks, the question
  bubbles — the renderer's output is byte-identical to pre-phase, so a
  document that legitimately contains a ` ```bor ` fence still renders its
  own text. Pass order with the option on:
  **verbatim → fences → tables → escape+inline → blockquotes → paragraphs →
  restore**.

- **Hydration paints the panel, and the panel is the file.** The client half
  is one classic script, `frontend/assets/verbatim.js`, loaded after
  `markdown.js` on the chat page and the shared page. At `done` (and on every
  restore) it matches each placeholder's normalized `data-bor-locator`
  against the frame's `answer_text[].locator` with a plain string compare —
  `markdown.js::borLocator` stays the ONLY locator grammar in the client, so
  the two sides cannot drift — and paints:

  - `.verbatim-caption` — `source/path · lines 12–40 of 210`, or
    `source/path · all 210 lines` when the block is the whole document: the
    numbers are the **server's** extent, so a locator whose range the model got
    wrong still shows what was actually read (a panel matched on `source/path`
    alone is A2 made visible).
  - `.verbatim-badge` — the word `VERBATIM`: these bytes came from the
    knowledge base, not from the model.
  - `.verbatim-lines` — the server's line numbers, `user-select: none`, so a
    selection or a Copy never drags digits into the clipboard. A capped block's
    marker line gets a BLANK cell, because the marker is not a line of the file.
  - `.verbatim-text` — `answer_text[].text` through `textContent`, byte for
    byte (never re-wrapped, never trimmed, never joined).
  - `.verbatim-copy` — `Copy` → `Copied` (the state is TEXT), accessible name
    `Copy N lines from <source/path>`; the clipboard receives the server string
    exactly.
  - `.verbatim-open` — the document viewer: on the chat page the same same-page
    modal as a source chip (back → the chat), on the shared page a plain
    `/document.html?source=…&path=…` link.
  - `.verbatim-more` — on a capped block, one muted line saying the rest is in
    the document, plus that link.

  A locator the server did not resolve renders the honest sentence *I could
  not verify this against the knowledge base — open `<source/path>`* with NO
  badge, NO body and none of the model's bytes; and every terminal path (the
  `error` frame, the 300 s guard, Stop, the failed-turn funnel) settles any
  leftover placeholder, so nothing ever sits at *Retrieving the exact text…*
  once a turn is over. A `show_text` block the answer never located is
  appended in the `answer_image` band — below the answer text, above the
  source clusters.

- **Every settled answer code block can be copied.** `enhanceAnswerCodeBlocks`
  wraps each `<pre>` inside a settled answer bubble that is not part of a
  panel in `.code-block` and adds a `.code-block-copy` pill: **no badge and no
  caption**, because a `VERBATIM` claim belongs only to a server-resolved
  block — an answer's code is either verbatim-and-badged or plainly the
  model's, never a silently-substituted hybrid. Buttons appear when the turn
  settles, never mid-stream. The share link's copy path delegates to the same
  `copyTextWithFallback`, so the http-origin inline select-to-copy degradation
  exists once (the owner-locked share strings and behavior are unchanged).
- **Parity, and what stays untouched.** The live turn, a reload and an
  anonymous shared page call the same three globals in the same order, so one
  answer renders one way. Only the two answer renderers (plus the shared
  page's) opt in: a question, a thinking run and the document viewer stay
  one-argument calls, and `document.html` does not even load the panel client.

---

## PDF documents (phase 126)

PDFs are **opt-in**: add the `pdf` token to `BOR_IMPORT_EXTENSIONS`
(a stock `.env` never indexes them). Once admitted, a PDF is a
first-class document through every entry point (git/local walks and
uploaded archives):

- **Text, page-aware.** Text is extracted page by page (`pymupdf`); one
  document per PDF, its text-bearing pages joined with `---PAGE-N---`
  separators. Chunking is page-aware — every chunk keeps a `Page N:`
  anchor, so a retrieval hit stays readable in context.
- **Summarized + dated like any document.** The `lite` model summarizes
  it, sync sources its date (git commit / file metadata), and it appears
  in the catalog, the KB tree, and the agent's `ls` / `read` / `grep`
  tools.
- **Embedded images join the image pipeline.** With `BOR_IMAGES=1`,
  images embedded in the PDF are extracted and run through the same
  image-document flow as standalone images (chat-model description,
  persistent copy under `BOR_IMAGE_DIR`, retrievable + displayable);
  with the toggle off the pass is skipped and the PDF text indexes
  normally. Embedded images in a format outside `BOR_IMAGE_EXTENSIONS`
  are skipped with a warning. The extraction is de-duplicated and
  viewer-faithful (phase 151): repeated placements of the same image
  yield ONE document (first placement wins), an image with a soft
  mask (`/SMask`) is stored as the composited image a PDF viewer
  shows (alpha PNG — not the blank base stream), and near-white,
  nearly-uniform images (the table-shading "blank white" underlays)
  are skipped entirely — no row, no copy, no vision call. The run
  counts the drops log-only on the import summary line
  (`pdf_images_deduped=N` `pdf_images_blank_skipped=N`).
- **Document context (phase 137).** Every embedded image is described
  *with* the document it came from: the vision prompt — and the stored
  `image_context` + enriched content — carry an
  `Extracted from <source>/<pdf> (page N):` origin line (always
  present, even for a text-less page) plus the page's own text
  (capped at 6 000 chars; dropped when the phase-133 gate flags the
  text layer as garbage — origin line only). Markdown `![alt](path)`
  references to the image's synthetic path ride along after it
  (phase 125, document block first), so the description names the
  document and the page the image illustrates. Image docs a
  pre-phase-137 sync left without the block (`image_context` NULL or
  markdown-only) are **backfilled on the next sync** — re-described
  once with the context (the parent's page text is re-read from the
  source while the PDF is still there; parent gone → origin line
  only), idempotent (the following sync burns zero vision calls for
  them), fail-soft (a failure leaves the row untouched and retries
  next sync), counted log-only on the import summary line
  (`image_context_backfilled=N`). The stored context is also VISIBLE
  in the chat suggestion block: an image doc's `<document>` body leads
  with a one-line locator — `From <source>/<pdf> (page N)` for a
  PDF-embedded image (with `; Referenced in …` appended when markdown
  reference blocks are stored too — document block first),
  `Referenced in <source>/<path>: "alt"` for a markdown-referenced
  image (phase 125) — the page-text body is never rendered.
- **Garbage text layer → page-screenshot fallback (phase 133).** Before
  indexing, the extracted text passes a deterministic garbage gate
  (empty text layer, or ≥ 30% tag-looking lines, or ≥ 5% non-printable
  chars — the class of PDF that reads as raw XML/binary soup). On
  failure every page is rendered to a screenshot and read by the vision
  model; the stored content is the `---PAGE-N---`-anchored page
  descriptions (a failed page gets a `page N — description unavailable`
  placeholder — fail-soft, the doc still lands). Clean text layers
  never burn a fallback vision call. The run counts the affected PDFs
  on the import summary line (`pdfs_screenshotted=N`). A garbage-text
  PDF takes this fallback instead of the pictorial pass below — the
  precedence is locked, never double vision.
- **Pictorial pages get vision-described (phase 140, on by default).**
  For a PDF whose text layer is usable but thin — the IKEA
  assembly-instruction class (20 of 24 pages: 4–73 chars of markers
  around vector-drawn pictograms) — every page whose STRIPPED text is
  under `BOR_PDF_PICTORIAL_MIN_CHARS`
  characters (default `100`; a deterministic per-page threshold, no LLM
  in the gate) is rendered to a PNG and described by the chat model,
  and the description appends to that page's content as
  `[Page N illustration: …]` — so the embeddings carry what the thin
  text layer couldn't, and a question about the illustrations hits.
  NO page cap: every candidate page is described (A21 — the LLM calls
  are free; a slower sync is acceptable). A PDF with zero candidate
  pages burns no render or vision call — its indexed content is
  byte-identical to pre-phase-140. The run counts the affected PDFs and
  pages on the import summary line (`pdfs_pictorial=N`
  `pdf_pictorial_pages=N`). `BOR_PDF_PICTORIAL=0` turns the whole pass
  off (byte-identical pre-phase-140 content).
- **Served as text — or as pages (phase 148).** The document viewer
  renders the extracted text — never the binary PDF. The exception is
  the two page-qualifying classes: a PDF whose text layer fired the
  phase-133 screenshot fallback, or a PDF whose phase-140 pictorial
  pass described ≥ 1 sparse page. For those, every page is rendered
  to a persistent PNG at index time (under `BOR_PDF_PAGE_DIR`) and
  the viewer — the doc modal AND `/document.html`, one shared core —
  shows the PAGES: a vertical strip of lazy page images with
  "Page N" captions, in place of the text content and the Summary
  panel (the per-page descriptions stay in the DB for Q&A — the
  embeddings and summary are untouched, just not displayed). Since
  phase 163 this strip-only shape is the PRIMARY class only — the
  pictorial class keeps its text behind a Text/Pages toggle, per the
  next bullet. A lost
  copy shows a "Page N unavailable" note for that page only. Text-
  heavy PDFs (usable layer, zero pictorial candidates — and the whole
  sparse class while `BOR_PDF_PICTORIAL=0`) keep the text view,
  byte-identical to pre-phase.
- **Which kind of strip it is (phase 163).** The two qualifying classes
  are not the same problem: the screenshot class has no usable text to
  show, while a book with one sparse page (an illustration map page in a
  36-page text) does. The document row therefore records the strip's
  CLASS beside the page count (`documents.page_strip_mode` —
  `primary` = the screenshots ARE the content, so the strip is the view;
  `additive` = a usable text layer, so the text + Summary view stays and
  the pages sit behind a Text/Pages toggle), and the document-content
  payload carries it as `page_strip_mode`. The key is ABSENT — never
  `null` — when there is no strip, which is also how a row written before
  this phase serializes; the viewer treats an absent class as `primary`,
  so every existing document renders exactly as before until the next
  sync stamps it (the sync's self-heal pass, phase 163).
- **The strip repairs itself (phase 163).** An unchanged PDF is not
  assumed to still have its page images on disk. On every sync, a row
  that claims a strip is re-gated deterministically from the current
  file bytes and the current settings — the same gate the changed path
  uses (`BOR_PDF_PICTORIAL` and `BOR_PDF_PICTORIAL_MIN_CHARS`
  included), with **zero vision or LLM calls** (rendering is CPU-only):
  the class is re-stamped, any `<doc-id>-p{N}.png` missing from
  `BOR_PDF_PAGE_DIR` is re-rendered (write-before-delete, atomically per
  file), and a strip that no longer qualifies — the kill switch turned
  off, the sparse-page bar lowered past it — is cleared along with its
  copies. So a wiped pages directory, or a copy set lost to an
  interrupted sync, repairs itself on the next sync instead of leaving
  "Page N unavailable" forever. It logs one `PDF strip re-gate …` line
  only when it actually acts, and a failure there never fails the sync.
  PDFs without a strip pay nothing extra (no extra read at all).

---

## Caching & deploys

A deploy is a commit — and the browser must see it **without a hard refresh**.
The first three bullets below are that A19 rule, applied at the transport layer
by `app/core/caching.py`; the last two are phase 167 — gzip from
`app/core/compression.py` and per-route validators from `app/core/http_cache.py`.
Outermost to innermost, the stack is security-headers → gzip (only when
`BOR_COMPRESS_MIN_BYTES > 0`) → the byte-budget logger → caching → routes, so
security headers land on every response and `bytes=` in the budget line is the
route's own, uncompressed payload (see *Operational notes*):

> **Security headers (A20).** Every response carries the app-wide
> `Content-Security-Policy` (`default-src 'self'` plus the phase-123
> `img-src 'self' data:` carve-out), `X-Frame-Options: DENY` and
> `X-Content-Type-Options: nosniff` — and the middleware *preserves* a
> CSP an inner layer already set. One route uses that deliberately: SVG
> document bytes (phase 181) ship `default-src 'none'; sandbox` instead,
> because a directly-navigated stored `.svg` must never become a
> script-capable same-origin document. Raster/PDF image responses gain
> nothing — their header set is unchanged.

- **HTML pages** ship `Cache-Control: no-cache`, **no `etag`**, **no
  `last-modified`** — each visit always gets a fresh 200 body.
- **Assets are versioned and cached for a year.** Pages reference their
  CSS/JS with a token (`/assets/styles.css?v=<token>`), and every
  `/assets/*` response ships `Cache-Control: public, max-age=31536000, immutable`.
- **The token is the deploy.** In a git checkout it is the short SHA of
  `HEAD` — so every commit/deploy flips the token and the versioned asset
  URLs change with it. A checkout without `.git` falls back to a stable
  content hash of the `frontend/` tree.
- **Responses are gzipped (phase 167).** Every HTTP response except the
  `POST /api/chat` SSE stream goes through selective gzip
  (`app/core/compression.py`): when the client accepts gzip and the body
  is at least `BOR_COMPRESS_MIN_BYTES` (default `860`; `0` = the feature
  is OFF), KB JSON ships ~8-12x smaller — the fix for the ~2 MB
  `/api/docs/tree` load. The SSE stream is skipped by EXACT path (A15:
  frames keep arriving one by one, uncompressed), small responses stay
  identity, and an already-encoded body is never double-encoded, so a
  fronting nginx may gzip too.
- **Catalogue reads revalidate (phase 167).** `GET /api/docs`,
  `GET /api/docs/tree` and `GET /api/docs/tree/node` ship a content-exact
  weak `ETag` (SHA-256 of the served body, `app/core/http_cache.py`) with
  `Cache-Control: private, must-revalidate` — a repeat load costs a 304
  with no body, and a sync or an admin edit (folder summary, document
  date) changes the body, hence the tag, so it is seen on the next load.
  The tag is deliberately NOT `sources_meta.version` (bumping that marks
  saved chats stale, and those edits never bump it). This is a scoped
  extension of the rule above: validators live on these three read-only
  routes only — HTML pages keep publishing none, and the SSE stream and
  every mutating route are untouched.

No CDN, no new services, no build-step change: the middleware rewrites the
asset references of the known pages in flight.

> **Deploy note:** the very first deploy onto this scheme needs one normal
> page visit, so the browser revalidates the HTML once and starts requesting
> the versioned assets; every commit after that is picked up automatically.

### Operational notes: the response byte budget (phase 167)

Every `/api/*` response writes ONE parseable line, so "which endpoint is
heavy, right now?" is answerable from the app's own log:

```
2026-10-08 07:40:57 INFO     app.response_budget :: api method=GET path=/api/docs/tree status=200 bytes=8474 enc=gzip duration_ms=12
```

```bash
grep ' :: api ' server.log                    # every measured response
grep ' :: api ' server.log | grep 'bytes=[0-9]\{6,\}'   # the six-figure ones
```

- **`bytes=`** is what the ROUTE produced (uncompressed), **`enc=`** is what
  went on the wire (`gzip` or `identity`) — "how fat is it, and how fat did
  the client see it". A revalidated `304` logs `bytes=0` on purpose: that is
  the win you want to see.
- **Level:** `INFO` at or over `BOR_LOG_LARGE_RESPONSE_BYTES` (default
  `65536`), `DEBUG` below it, so the default log stays readable.
  `BOR_LOG_LARGE_RESPONSE_BYTES=0` logs EVERY `/api/*` response at INFO —
  the mode for a one-off investigation. A negative value refuses startup.
- **`stream=1`** marks the SSE turn (`POST /api/chat`, A15): `bytes` is the
  streamed total and `duration_ms` the stream's life. One line per stream,
  written when it ends — including a stream the client cancels (with the
  bytes that got out). The middleware is pure ASGI and never buffers.

**Measuring the whole API** (the audit behind the fix):

```bash
# the corpus that matches production's order of magnitude
#   (destructive to the local DB — see the restore-test-db skill)
uv run uvicorn app.main:app --port 8000
uv run python -m scripts.api_payload_audit --base-url http://localhost:8000
```

It signs in with `BOR_ADMIN_PASSWORD` (or `--password`; the single-admin
login has no username), measures every candidate read route in BOTH
`identity` and `gzip` forms plus one conditional revalidation, prints the
corpus it measured, and marks anything over `--budget` (default `262144`)
with `LOOK`. It is **read-only** (safe against the deployed app) and always
exits 0 when it got its numbers — it is a report, not a gate (LOCKED A5:
nothing in this phase rejects a large payload). Exit `2` means the app was
unreachable or the login was rejected. The measured before/after tables and
the verdict per endpoint live in the phase record — `167_api_payload_diet/AUDIT.md`
under `.agents/phases/`.

> **Deploy note (compression):** the app gzips its own responses, so nginx
> `gzip on` is **optional** — leaving it off ships the same bytes, and
> leaving it on is fine too: an already-encoded body is never double-encoded
> by either side. Passing the client's header through untouched
> (`proxy_set_header Accept-Encoding $http_accept_encoding;`, or simply not
> overriding it) is correct; unsetting it would hand every client
> uncompressed JSON.

### Operational notes: the sync's memory (phase 187)

The 2026-10-15 incident made "how much RAM did that sync use?" a dig
through `journalctl` unit-teardown lines from three boots — and the boot
that killed the box had no teardown record at all, because it never shut
down cleanly. Every sync lifecycle line now carries the number, so the
question is one grep:

```
2026-10-15 04:12:07 INFO     app.api.sync :: sync: done detail={'files': 27411, …} rss_mb=412 rss_peak_mb=1936
```

```bash
grep ' :: ' server.log | grep ' rss_peak_mb='          # every sync terminal
grep ' :: ' server.log | grep 'sync: started '         # where each run began
grep ' :: ' server.log | grep 'sync: failed state='    # the failed ones, with the error
```

- **`rss_mb=`** is the process's CURRENT resident set size in MB (Linux:
  `VmRSS` from `/proc/self/status`), read at the moment that line is
  written — the run's start, the import's summary line, and every
  terminal.
- **`rss_peak_mb=`** is the process HIGH-WATER mark
  (`getrusage(RUSAGE_SELF).ru_maxrss`): the most the process ever held,
  which is the number the incident was actually about. It appears on the
  three terminals (`sync: done …`, `sync: cancelled by user …`,
  `sync: failed state=failed …`) beside the final counts. The two numbers
  come from different kernel counters and both floor to whole MB, and the
  kernel refreshes the high-water **lazily**, so a raw `rss_peak_mb()` can
  sit under a raw `rss_mb()` (measured: 164 against 162). The run's own
  sampling point (`app.api.sync._sample_rss`) therefore lifts a lagging
  peak to the current reading before anything publishes it — so on every
  sync line, on `/api/sync/status` and on the page you always get
  `peak >= current`. An unknowable peak (`?` / `null`) is never fabricated
  from the current reading.
- **`?` means "unknowable here", never 0 and never `None`** —
  `app/core/memory.py` never raises, so a platform without `/proc` (or
  without `resource`) degrades the reading and the sync keeps running.
- **Process vs cgroup — know which number you are reading.** These are
  PROCESS numbers. The host's `/sys/fs/cgroup/memory.peak` — and the
  `systemd` "… memory peak" line the incident started from — is a
  different, larger figure, because a cgroup counts **page cache** too:
  that box's app cgroup showed ~2.16 GB of `file` against ~211 MB of
  `anon` while idle. Neither figure is wrong; the log says which one it
  is (LOCKED A1), and `app.core.memory.cgroup_peak_mb()` exists for an
  operator who wants the other one for context.
- **Append-only** (LOCKED A2): the fields trail the existing ones on the
  pre-phase lines (`sync: started repos=… origin=… git=… local=…`,
  `sync: done detail=…`, `sync: cancelled by user state=cancelled
  files_done=… files_total=…`, `import: summary files=… … formats=…`),
  and the failed funnel keeps its `logger.exception` traceback while
  gaining ONE sanitized one-line summary (`error=` carries exactly the
  masked string the status field does). So every grep written before this
  phase still matches.

- **On the page, not only in the log.** The same pair rides
  `GET /api/sync/status` (admin-only, like every sync surface) as the
  additive `rss_mb` / `rss_peak_mb` keys, and the RAG page
  (`/sources.html`) appends them to the result line it already speaks —
  `3 added · 1 unchanged · peak 1936 MB`, or
  `Sync cancelled · peak 1936 MB`. A terminal KEEPS
  the final reading next to the final counts, so a page reloaded an hour
  later still shows what that run reached, and `sync: done …` logs the
  very same numbers (one reading, two surfaces — the page and the log
  can never disagree).
- **Sampled, never per-file** (LOCKED A4): the run reads its own memory
  at the run start, at each phase transition (`import` → `overview` →
  `summaries`), in every terminal, and — throttled to at most one read
  per 5 s — from the import's per-file progress hook, so the number
  moves while you watch a long walk without costing 27k `/proc` reads.
- **`null` renders NOTHING.** No reading (idle, a run that predates this
  phase, the upload job's status, a platform that cannot know) shows no
  clause at all — never `peak ? MB`, never `peak 0 MB`. The failed
  terminal's box stays empty by the existing contract (the banner and
  the modal carry the error, byte-identical); a failed run's numbers are
  in its log line and in the status payload.

There is deliberately **no** memory audit script and **no** metrics
endpoint (LOCKED A3/A2 — no aux services, A12): the grep above is the
interface.

---

## Configuration reference

Everything is an environment variable (the `BOR_` prefix, plus
`DEBUGPY*`) read from `.env`. **Every variable in
[`.env.example`](.env.example) is documented below** — copy that file
to `.env` and adjust.

**DB-over-env overrides (phase 168):** the precedence for every setting
is **`app_settings` row → `BOR_*` environment → code default** — an
override the admin saves in the Settings panel is stored in Postgres
(`app_settings`, one row per key) and **wins over the env var for
that key — with no restart**. The panel never writes `.env` (the env
value stays the base; per-key Reset deletes the row and the env value
wins again). `BOR_SETTINGS_DB=0` (set in the **container env**, not
`.env` — a committed `0` is a silent "my change did nothing")
disables all DB overrides — the pre-phase behavior. Two keys are NOT
overridable: `BOR_DATABASE_URL` (it builds the very engine that would
read the override) and `BOR_EMBEDDING_DIM` (fixed at `chunks` table
creation). A stored value that fails the setting's validators is
skipped at read with a logged reason — a DB row never bricks boot.
"No restart" is not universal: the settings the panel marks
**restart required** (DB pool sizes, session cookie wiring, log
level, compression, the static dir) apply on the next
`podman compose restart app`; everything else applies immediately.
Admins read/write the overrides through the admin-only
`/api/admin/settings` routes (GET the catalogue, PUT `/{key}` set one
value, DELETE `/{key}` reset one, DELETE all clear everything);
secrets are write-only there (the response reports `has_value`,
never the plaintext). A `docs_repo` PUT with embedded credentials
(`://user:pass@` — the phase-173 leak class) is refused with a 422
pointing at the `docs_user` / `docs_token` fields; a legacy `.env`
value of that shape is split into those two settings once at
startup (the DB rows shadow the env, `.env` is never rewritten).

### App & branding

| Env | Default | Meaning |
|-----|---------|---------|
| `BOR_APP_NAME` | `Brain of Reese` | Display name everywhere (page titles, header brand, status labels) |
| `BOR_APP_VERSION` | `0.1.0` | Version the FastAPI app reports (`GET /api/health`, the OpenAPI doc) |
| `BOR_INPUT_PLACEHOLDER` | `Ask me anything…` | Chat composer placeholder |
| `BOR_FOOTER_TEXT` | `Powered by self-hosted models` | Footer line on every page |
| `BOR_ENVIRONMENT` | `development` | `development` or `production` (the prod compose profile sets `production`) |
| `BOR_LOG_LEVEL` | `INFO` | App log level |
| `BOR_STATIC_DIR` | `frontend` | Where static files are served from (the container image sets `/app/static`) |
| `BOR_COMPRESS_MIN_BYTES` | `860` | Selective-gzip floor (phase 167): responses ≥ this size compress when the client accepts gzip; `0` = OFF (no middleware), negative refuses startup. `POST /api/chat` (SSE) is never compressed |
| `BOR_LOG_LARGE_RESPONSE_BYTES` | `65536` | INFO/DEBUG cut for the per-response `api method=… bytes=… enc=…` byte-budget line (phase 167, `grep ' :: api '`); `0` = log every `/api/*` response at INFO, negative refuses startup. See *Operational notes* |

### Database & concurrency

| Env | Default | Meaning |
|-----|---------|---------|
| `BOR_DATABASE_URL` | local compose URL | SQLAlchemy URL (psycopg) |
| `BOR_DB_POOL_SIZE` | `5` | DB connection pool size |
| `BOR_DB_POOL_MAX_OVERFLOW` | `10` | Overflow connections above `BOR_DB_POOL_SIZE` |
| `BOR_DB_POOL_RECYCLE` | `3600` | Recycle pool connections after N seconds |
| `BOR_CHAT_MAX_CONCURRENT` | `10` | Max concurrent `/api/chat` turns |

### LLM (aipi)

| Env | Default | Meaning |
|-----|---------|---------|
| `BOR_LLM_BASE_URL` | `https://aipi.reeseapps.com/v1` | OpenAI-compatible endpoint |
| `BOR_LLM_API_KEY` | — (falls back to `$AIPI_KEY`) | aipi API key |
| `BOR_LLM_CHAT_MODEL` | `turbo` | Chat model — must support vision when `BOR_IMAGES=1` |
| `BOR_LLM_EMBED_MODEL` | `embed` | Embedding model |
| `BOR_LLM_SUMMARY_MODEL` | `lite` | One-shot completions: document summaries at import, KB overview, folder summaries |
| `BOR_LLM_CHAT_THINKING_LEVEL` | `low` | The chat turn's thinking level (pi-style, free-form — no allowlist); rides the streaming chat-completions requests as `reasoning_effort`; empty = the parameter is not sent |
| `BOR_LLM_SUMMARY_THINKING_LEVEL` | *(empty)* | The one-shot summary thinking level (document/folder summaries, KB overview, the model probe); rides the one-shot `chat()` calls as `reasoning_effort`; empty (default) = the parameter is not sent |
| `BOR_EMBEDDING_DIM` | `768` | Vector dimension (fixed at table creation) |
| `BOR_LLM_RETRIES` | `3` | Retries: chat-turn stream before the first token + empty one-shot summary replies; `0` = off |
| `BOR_LLM_RETRY_DELAY` | `5` | Seconds between LLM retries |
| `BOR_LLM_TIMEOUT` | `120` | HTTP timeout (s) for LLM API calls |
| `BOR_STREAM_THINKING` | `1` | Stream the model's thinking as `thinking` SSE events (`0` = suppress) |

### RAG & prompt budgets

| Env | Default | Meaning |
|-----|---------|---------|
| `BOR_TOP_N_DOCS` | `2` | Ceiling for the cited tier, not a quota (phase 113) — a single strong doc yields one citation |
| `BOR_RELEVANCE_THRESHOLD` | `0.62` | Answer when best cosine ≥ this **or** a lexical hit corroborated by cosine ≥ `BOR_LEXICAL_SUPPORT_FLOOR`; otherwise honest deflection |
| `BOR_LEXICAL_SUPPORT_FLOOR` | `0.35` | Best-cosine floor an FTS hit must clear to flip the gate HIGH (A8 revised 2026-09-14) |
| `BOR_SOURCE_USEFULNESS_FLOOR` | `0.35` | Cosine bar for a citation slot (phase 113): a doc earns the cited tier only when its best-chunk cosine clears it; `0` = no bar |
| `BOR_RELATED_MAX_DOCS` | `2` | Cap on the related-doc tier (the de-emphasized "nearby docs" row — never a citation chip); `0` = no related docs |
| `BOR_SUGGESTED_DOCS` | `5` | Cap on the "start here" suggestion tier (phase 118): top-N doc summaries seeded into the grounded prompt; no cosine floor; ≥ 1 |
| `BOR_SUGGESTION_PREVIEW_CHARS` | `400` | NULL-summary fallback preview in a suggestion block (phase 118): first N content chars + truncation marker; > 0 |
| `BOR_HISTORY_MAX_TURNS` | `40` | Prior client-provided history turns kept per chat turn (phase 74, newest first); `0` = no history (the pre-phase-74 two-message requests) |
| `BOR_HISTORY_MAX_CHARS` | `24000` | Total char budget for the kept history — `len(text) + len(thinking)` per turn (phase 74); a turn that would overflow is dropped whole, never cut mid-answer; `0` = no history |
| `BOR_MAX_OUTPUT_TOKENS` | `32768` | Max answer length in tokens (answers must not be cut off) |
| `BOR_STEERING_MAX_CHARS` | `8000` | Char budget for the `<tuning>` (steering notes) prompt section |
| `BOR_SUMMARY_MAX_CHARS` | `12000` | Cap on document content sent to the `lite` summary model |
| `BOR_KB_OVERVIEW_MAX_CHARS` | `4000` | Char budget for the `<knowledge_base>` prompt section — the stored outline **plus** its deterministic `Sources: …` tail (phase 162); the cap keeps the longest-fitting prefix, so a value below the default clips the tail first (the default always holds both) |
| `BOR_OVERVIEW_INPUT_MAX_CHARS` | `40000` | Cap on the document list sent to `lite` for KB overview generation |
| `BOR_FOLDER_SUMMARY_INPUT_MAX_CHARS` | `8000` | Cap on a folder's document list sent to `lite` for folder summaries (phase 94) |

### Chunking & embedding

| Env | Default | Meaning |
|-----|---------|---------|
| `BOR_CHUNK_TARGET_CHARS` | `2000` | Target chunk size (chars) |
| `BOR_CHUNK_OVERLAP_CHARS` | `200` | Overlap between adjacent chunks (chars) |
| `BOR_EMBED_BATCH_SIZE` | `16` | Chunks per embedding API call |
| `BOR_EMBED_QUESTION_MAX_CHARS` | `1200` | Chat question-embed prefix cap (phase 114): the embed step embeds at most this many chars of the question — the full question still reaches the LLM prompt |

### Hybrid retrieval

| Env | Default | Meaning |
|-----|---------|---------|
| `BOR_HYBRID_VECTOR_CANDIDATES` | `100` | Cosine list width for the RRF fusion |
| `BOR_RETRIEVAL_EF_SEARCH` | `400` | HNSW `ef_search` for the cosine tier (phase 188) — `SET LOCAL` per transaction; clamped UP to `BOR_HYBRID_VECTOR_CANDIDATES` with a WARNING when lower (that clamp is the recall floor, so `0`/negative fails startup); raise it for more recall, Postgres caps it at 1000. 400 is the measured depth at which the graph recovers the exact top-100 (see [How retrieval works](#how-retrieval-works-hybrid)) |
| `BOR_HYBRID_LEXICAL_CANDIDATES` | `30` | FTS list width for the RRF fusion |
| `BOR_RRF_K` | `60` | RRF damping constant (`1/(k + rank)`) |
| `BOR_RECENCY_BOOST` | `0.0007` | Max additive fused-score bonus a zero-age doc gets (phase 106); `0` = off |
| `BOR_RECENCY_HALF_LIFE_DAYS` | `365` | Recency decay timescale, days (phase 106) — the boost decays by e^-1 (~0.37x) per timescale of age |
| `BOR_NAME_HIT_BONUS` | `0.005` | Selection-time bonus a name-hit doc's best fused score gets (phase 119); `0` = off |

### Agent tools

| Env | Default | Meaning |
|-----|---------|---------|
| `BOR_AGENT_MAX_ROUNDS` | `10` | Hard cap on agent tool rounds per grounded turn (0 = no tools); every tool result carries the model's remaining count (phase 160) |
| `BOR_READ_MAX_CHARS` | `128000` | Cap on a `read` tool result (chars); over-cap reads truncate + a notice that teaches the grep → `read(offset, limit)` workflow (phase 95, rewritten phase 131) |
| `BOR_ANSWER_TEXT` | `1` | Verbatim file display (phase 182): the **master switch, ON by default** — an answer's code block is typed by the server from the indexed bytes, never retyped by the model. `0` = the escape hatch: the `show_text` tool is not offered, its `<tools>` teaching clause is absent, ` ```bor ` locator fences are not resolved, and the `done` frame carries no `answer_text` key (the server surface is exactly pre-phase; a stray fence on the client still settles to the honest unresolved sentence rather than the model's copy) |
| `BOR_SHOW_TEXT_MAX_CHARS` | `12000` | Per-block character budget for ONE displayed verbatim slice (phase 182): the slice is cut at a **line boundary**, the shared `[…truncated…]` marker is appended and the caption's line range shrinks to what is actually shown. `0` = inert (nothing displays); a negative value refuses startup naming the field |
| `BOR_SHOW_TEXT_MAX_BLOCKS` | `5` | Per-answer cap on displayed verbatim blocks (phase 182), counted in answer order. `0` = inert (nothing displays); a negative value refuses startup naming the field |

### Import & sources

| Env | Default | Meaning |
|-----|---------|---------|
| `BOR_IMPORT_EXTENSIONS` | A9 family | Importable text formats — the A9 default is `md,markdown,txt,yaml,yml,json,py,container,network,volume,image,pod,kube,swap,os,endpoint,j2`; any well-formed extension is allowed. Add the `pdf` token to make PDFs first-class, retrievable documents (phase 126, opt-in). Images are **not** added here — see `BOR_IMAGES` |
| `BOR_PDF_PICTORIAL` | `1` | Master switch for the pictorial PDF pass (phase 140, on by default): a usable-text PDF's sparse pages (stripped text under `BOR_PDF_PICTORIAL_MIN_CHARS`) are rendered + vision-described into the indexed content (`[Page N illustration: …]`); `0` ⇒ content byte-identical to pre-phase-140 (no render, no vision call) |
| `BOR_PDF_PICTORIAL_MIN_CHARS` | `100` | Sparseness bar for the pictorial pass (phase 140): a page whose STRIPPED text is STRICTLY shorter than this many chars is a candidate (a page with exactly this many is not); `0` = every page (A21); a negative value fails startup loudly |
| `BOR_PDF_IMAGE_SKIP_BLANK` | `1` | Skip blank/decorative images embedded in PDFs (phase 151, on by default): near-white, nearly-uniform images (the table-shading "blank white" underlays) are dropped from the embedded-image pass — no row, no copy, no vision call; `0` ⇒ every extracted image is indexed again (pre-phase behavior; dedupe + soft-mask compositing unaffected) |
| `BOR_PDF_PAGE_DIR` | `~/bor-sources/pages` | Persistent home for the rendered PDF page PNGs (phase 148): a page-qualifying PDF (the phase-133 screenshot class or the phase-140 pictorial class) gets every page rendered at index time as `<doc-id>-p{N}.png` here, so the document viewer shows the actual pages (the descriptions stay embedded for Q&A, just not displayed); text-heavy PDFs keep the text view. Clearing the directory is self-healing — the next sync re-gates any strip that still claims pages and re-renders what is missing, with zero vision calls (phase 163) |
| `BOR_GIT_SOURCES` | — (empty) | CSV of git repo URLs — **fallback while the admin Git sources page's list is empty** |
| `BOR_SOURCES_DIR` | `~/bor-sources` | Where git repos are cloned/pulled |
| `BOR_UPLOAD_DIR` | `~/bor-sources/uploads` | Where uploaded source archives are unpacked |
| `BOR_UPLOAD_MAX_MB` | `512` | Cap (MiB) for uploaded source archives (zip-bomb guard) |
| `BOR_SYNC_SESSION_RESET_FILES` | `200` | Files between releases of the import run's SQLAlchemy session (phase 184): every Nth file the walk hands back everything the session still holds (`expunge_all`), so an unchanged-file sync stops keeping one full document row per file walked. `0` = never release — the exact pre-phase-184 behaviour; a negative value refuses startup naming the field. The release also runs once per source, before the two post-passes |
| `BOR_SYNC_PRUNE_BATCH` | `500` | Pruned documents per COMMIT inside the prune (phase 184) instead of one commit at the end of the whole import, so a mass prune releases its deletes as it goes instead of holding one enormous transaction. `0` = one commit at the end — the exact pre-phase-184 behaviour; a negative value refuses startup naming the field. With a non-zero batch a crash mid-prune leaves a **prefix** pruned, and the next sync converges (`seen` is recomputed from disk every run) |
| `BOR_SUGGESTIONS` | built-in list | JSON seed for onboarding chips (shown before the first saved question) |

### Image support (phases 122/123/124/181)

| Env | Default | Meaning |
|-----|---------|---------|
| `BOR_IMAGES` | `0` | Master switch for image support — **off by default; enable only when `BOR_LLM_CHAT_MODEL` supports vision**. Gates both image documents and question images |
| `BOR_IMAGE_EXTENSIONS` | `png,jpg,jpeg,webp,gif,bmp,svg` | Extensions indexed as image documents (case-insensitive; separate from `BOR_IMPORT_EXTENSIONS`). `svg` is the one **vector** member: indexed from its own `<title>`/`<desc>`/`<text>` with **no vision call**, served as `image/svg+xml` for a real preview |
| `BOR_IMAGE_DIR` | `~/bor-sources/images` | Persistent home for the served image bytes (uploads are replaced, checkouts re-cloned) |
| `BOR_CHAT_IMAGE_DIR` | `~/bor-sources/chat-images` | Where question images are stored (saved/shared chats carry the path, never base64) |
| `BOR_CHAT_IMAGE_MAX_MB` | `10` | Question-image upload cap (MiB; must be > 0) |
| `BOR_ANSWER_IMAGES` | `0` | Let the chat model show a standalone image document as part of its answer (phase 124): offers the agent the `show_image` tool and renders the shown image as an answer figure — **independent of `BOR_IMAGES`** (that toggle indexes image docs; this one shows them). The text-display sibling (`BOR_ANSWER_TEXT`, phase 182) lives in [Agent tools](#agent-tools) |

### Docs push

| Env | Default | Meaning |
|-----|---------|---------|
| `BOR_DOCS_REPO` | — (empty) | The git repo chat answers can be committed to (URL or local path); while empty "Save as doc" is hidden and the push endpoint 409s — a **clean URL only**: credentials in the repo URL are no longer supported (phase 173), use `BOR_DOCS_USER` / `BOR_DOCS_TOKEN`. A legacy `://user:pass@` value (env or a saved override) is auto-split on startup into those settings, one-time and idempotent — the `.env` file is never rewritten |
| `BOR_DOCS_USER` | — (empty) | The docs-push remote's username; travels to git as the `GIT_CURL_USERNAME` environment variable (never in the command line); empty = the push is anonymous (phase 173) |
| `BOR_DOCS_TOKEN` | — (empty) | The docs-push remote's token — **write-only**: the settings API reports `is_set` and never the value, and the admin panel renders it masked; travels to git as `GIT_CURL_PASSWORD` (with `GIT_TERMINAL_PROMPT=0`); empty = the credential env is not set at all (phase 173) |
| `BOR_DOCS_BRANCH` | `bor-docs` | Branch pushes land on (`git push --ff-only` — you open the PR yourself) |
| `BOR_DOCS_BASE_BRANCH` | `main` | Branch each push is cut fresh from |
| `BOR_DOCS_WORK_DIR` | `~/bor-docs` | Where `BOR_DOCS_REPO` is checked out on the server |

### Admin & debugging

| Env | Default | Meaning |
|-----|---------|---------|
| `BOR_ADMIN_PASSWORD` | *(required)* | The single admin's password — app refuses to start when empty |
| `BOR_SESSION_SECRET` | *(required)* | Signing key for the `bor_session` cookie |
| `BOR_SESSION_MAX_AGE` | `43200` | Session-cookie lifetime in seconds (12 h, sliding) |
| `BOR_SESSION_COOKIE` | `bor_session` | Name of the signed session cookie |
| `DEBUGPY` | `0` | `1` ⇒ attach-on-demand debugpy on `DEBUGPY_PORT` |
| `DEBUGPY_PORT` | `5678` | debugpy listen port |

### Customizing the look

Every identity string is an env var: `BOR_APP_NAME` (display name),
`BOR_INPUT_PLACEHOLDER` (chat composer placeholder), and
`BOR_FOOTER_TEXT` (footer line) — all served by `GET /api/config` and
applied by `assets/brand.js` at boot.

The colors are set from the admin **Theme tab** (`/theme.html`,
admin-only — phase 91): the 17 palette colors (the 9 identity colors plus
the 8 semantic state colors — success/error/accent families) and the
three strings above are edited with text fields and color pickers,
persisted in the `ui_settings` table, and injected into every served page
as an inline `<style>` BEFORE first paint (no red flash, no pop-in). The
browser-tab **favicon** follows the same effective palette: `GET
/favicon.svg` renders the hexagon art from the saved colors on every
request (no process cache — a theme change is visible on the next
request, no restart) and degrades to the built-in art with 200 on a
DB blip (never 500). In the Theme tab, a live (unsaved) pick of
`surface` / `brand` / `brand_ink` repaints the favicon in real time
(a CSP-legal `data:` URI), and Save / Reset / re-show restore the
`/favicon.svg` route — the settled palette always wins. The old
CSS-file theming is retired — a leftover `BOR_THEME` line in a
deployment's `.env` is simply ignored. Leave everything unset and the
app renders the built-in dark-tech palette byte-identically.

---

## Checking retrieval quality

Ask the *real* pipeline (live aipi embeddings + the current KB) whether a
question lands on the right document, with gate verdict and per-document
cosine / FTS / fused scores:

```bash
uv run python -m scripts.eval_retrieval "How did I install gitlab?"
uv run python -m scripts.eval_retrieval --from-file questions.txt --top 8
```

Requires an aipi key — `BOR_LLM_API_KEY` in `.env` (the stock
configuration) or `$AIPI_KEY` in the environment — and an imported
knowledge base.

---

## Debugging

`debugpy` is **off by default** and *never imported* unless you opt in:

```bash
DEBUGPY=1 uv run uvicorn app.main:app
# → log line: debugpy: remote debugging ENABLED, listening on 0.0.0.0:5678
```

Attach from VS Code — the repo ships `.vscode/launch.json` with the
matching configuration (`"Python Debugger: Remote Attach`), mapped for a
**local** dev run:
```json
{
  "name": "Python Debugger: Remote Attach",
  "type": "debugpy",
  "request": "attach",
  "connect": { "host": "localhost", "port": 5678 },
  "pathMappings": [
    { "localRoot": "${workspaceFolder}", "remoteRoot": "." }
  ]
}
```
Attaching to a run **inside the container** instead needs the container's
own root: `"remoteRoot": "/app"` (that is where the image puts the code).

The port is non-blocking and attach-on-demand: the app keeps running
normally until you attach. Override the port with `DEBUGPY_PORT`.

---

## QA / Testing

Three layers — the project rule is **one phase, one Playwright suite** (a
phase may implement a user story or an owner-directed change; see
`AGENTS.md`):

```bash
# Unit + integration (FastAPI TestClient)
uv run pytest

# Coverage gate (phases require >90% on app/)
uv run pytest --cov=app --cov-report=term-missing

# Lint + static types
uv run ruff check .
uv run pyright

# Playwright E2E — install the browser once:
uv run playwright install chromium

# Each phase's E2E runs IN ISOLATION — one file per run:
uv run pytest tests/e2e/test_import_documents.py -v --no-cov
uv run pytest tests/e2e/test_chat_rag.py -v --no-cov
# ...one file per phase in tests/e2e/ (the stories live in
# .agents/user_stories/, see .agents/phases/todo/).
#
# Isolation is REQUIRED, not a style preference: every suite shares ONE
# Postgres, one mock aipi port (8901 — E2E_MOCK_PORT) and the KB it seeds,
# and each file truncates + reseeds the KB for its own fixture. Running
# the whole `tests/e2e/` directory in a single pytest session interleaves
# those fixtures and fails tests that pass file by file.
#
# Ports: the shared session app (tests/e2e/conftest.py) binds 8123
# (E2E_APP_PORT). A suite that needs its OWN app env boots a per-module
# uvicorn on its own port (phase 79) — e.g. E2E_APP_PORT_PRUNECASCADE for
# tests/e2e/test_prune_cascade_cleanup.py — because a second uvicorn on
# the session port dies on bind and the suite would drive the wrong
# server. So "isolation" is about the shared DATABASE, not about ports.

# To sweep every suite, loop over the files (one pytest process each):
for f in tests/e2e/test_*.py; do uv run pytest "$f" --no-cov -q || echo "FAIL $f"; done
```

**`git` on `PATH` is an E2E prerequisite** (phase 28): the git-source
suites drive real `clone_or_pull` / `file_commit_dates` calls against
generated local `file://` fixture repos. The heaviest is phase 179's
`tests/e2e/test_sync_git_freeze.py`: it builds a ~150 MiB pack-free repo
(30 commits × 5 MiB of seeded pseudo-random bytes) so a fresh
`git clone` genuinely costs ≈ 9 s — that clone IS the timing fixture for
"a long clone must not freeze the server". It is mock-only
(`E2E_REAL_LLM=1` is skipped there: the clone, not the LLM import, is
what the suite times).

**Retrieval regression battery.** `tests/fixtures/retrieval_battery.txt` is
the tracked retrieval gate (phases 119 + 162): the name-your-tool questions,
the versioned-name (`qwen 3.8 27b`) regression, the out-of-KB deflection
sanity, and the `besta` / `BESTÅ` accent-fold pair. Run it against a KB with
`scripts.eval_retrieval` (the real pipeline, real embeddings):

```bash
uv run python -m scripts.eval_retrieval --from-file tests/fixtures/retrieval_battery.txt
```

The exact expected ranks are recorded in the fixture's header comments; a
question whose source is missing from the KB snapshot is recorded as skipped
loudly, never silently dropped.

**Deterministic E2E:** by default the E2E app talks to a local **mock
aipi** (`tests/e2e/mock_llm.py`) whose embeddings are real token-overlap
vectors — so the cosine relevance threshold behaves like production. To
run E2E against the **live** self-hosted models instead:

```bash
E2E_REAL_LLM=1 uv run pytest tests/e2e/test_chat_rag.py -v --no-cov
```

---

## Production Deployment

Build the multi-stage image (frontend minified by esbuild in the builder
stage, deps installed by `uv`, non-root runtime):

```bash
podman build -t brain-of-reese/app:latest .
```

Run standalone (bring your own Postgres + pgvector):
```bash
podman run -d --name brain-of-reese \
  -p 8000:8000 \
  -e BOR_DATABASE_URL=postgresql+psycopg://reese:SECRETPASSWORD@dbhost:5432/brain_of_reese \
  -e BOR_LLM_BASE_URL=https://aipi.reeseapps.com/v1 \
  -e BOR_LLM_API_KEY=$AIPI_KEY \
  -e BOR_ADMIN_PASSWORD=your-password \
  -e BOR_SESSION_SECRET=<the-hex-from-the-setup-above> \
  brain-of-reese/app:latest
```

`BOR_ADMIN_PASSWORD` and `BOR_SESSION_SECRET` are **required** — the
fail-loud boot gate in [Admin & Sign-in](#admin--sign-in) refuses to
start while either is empty.

The entrypoint runs `alembic upgrade head` automatically on start.

> **Deploy note (migration `0029`, phase 188):** this step builds
> `ix_chunks_embedding_hnsw` — an HNSW index over `chunks.embedding` — and
> it is the one migration in this repo that takes **minutes** on a full
> corpus and needs free RAM to do it. It will not lock the table, it can
> be interrupted, and it self-heals — but read
> [Deploying migration 0029 (HNSW)](#deploying-migration-0029-hnsw) before
> running it on the production box.

Or run the whole stack from compose (app + db):
```bash
podman compose --profile prod up -d --build
```
The `app` service picks up `BOR_ADMIN_PASSWORD` / `BOR_SESSION_SECRET`
from the shell environment (or an env file) — the same fail-loud gate
applies.

Production hardening: app runs as non-root (uid 10001), slim image,
healthcheck on `/api/health`, debugpy off unless `DEBUGPY=1`, all assets
served locally (no CDN), `BOR_ENVIRONMENT=production`.

### Deploying migration 0029 (HNSW)

Migration `0029` builds `ix_chunks_embedding_hnsw` — an HNSW graph over the
existing `vector(768)` column (`m = 16`, `ef_construction = 64`), so the
cosine tier of every chat turn stops sequential-scanning
[`benchmarks/phase188_live_gate/`](benchmarks/phase188_live_gate/README.md).
Everything changes about the **plan**; nothing changes about the **answer**
(the measured parity is in the benchmark link above and in
`tests/integration/test_chunks_hnsw_recall.py`). It is still the migration
to run deliberately rather than as a side effect of a deploy.

**Pre-flight — all four, on the machine that runs it.**

```bash
free -m                                     # free RAM, whole box
psql "$BOR_DATABASE_URL" -c "show shared_buffers;"       # + maintenance_work_mem
psql "$BOR_DATABASE_URL" -c "select extversion from pg_extension where extname='vector';"
podman exec brain-of-reese_db_1 df -h /dev/shm           # the one people miss
```

* **pgvector ≥ 0.5.0** (`0.8.0` measured). HNSW does not exist before 0.5 —
  the migration fails with `access method "hnsw" does not exist`.
* **Disk:** budget **~3.9 KB per chunk row** — the real ~414k chunks need
  roughly **1.6 GB** for the index on top of the ~3 GB heap. A build that
  runs out of disk is the one failure the self-heal cannot talk its way out
  of.
* **RAM:** the build itself is not the hungry part — a 416,702-row build
  peaked at **81 MB** of RSS in the `alembic` process at the container's
  default `maintenance_work_mem` (64 MB), and 1.6 GB of index is page cache
  afterwards, which Linux drops under pressure. What the box needs is room
  for the build to finish while the app keeps serving.
* **`/dev/shm`:** see the `maintenance_work_mem` note below — inside this
  project's compose container it is **63 MB**, and that is what caps the
  knob, not the box's RAM.

**Build it.**

```bash
uv run alembic upgrade head
```

`CREATE INDEX CONCURRENTLY` takes `ShareUpdateExclusiveLock`: reads and
writes continue, the app keeps answering through the whole build, and the
build takes **~161 s for 416,702 vectors** at `maintenance_work_mem = 64 MB`
(measured, `production_order_build.log`) — budget minutes, not seconds, and
more on a 4 GB-class box.

*On `maintenance_work_mem`:*

* Never rely on the PostgreSQL **4 MB** default if you can set the knob at
  all — a 4 MB graph build spills and is dramatically slower.
* **But do not raise it above `/dev/shm` in this container.** D5's original
  suggestion (`SET maintenance_work_mem = '512MB'`) was run at production
  order and failed:
  `could not resize shared memory segment "/PostgreSQL.3366409952" to 533761504 bytes: No space left on device`
  — the container's `/dev/shm` is **63 MB**, and the dying build leaves an
  **INVALID** index behind (`build_maintenance_work_mem_512MB_failure.log`).
  The migration's self-heal then did its job (dropped the INVALID graph,
  rebuilt it valid), so nothing was lost, but the recorded procedure for
  THIS container is: **`alembic upgrade head` with the container default
  (64 MB)**, which costs time and not correctness. Raising
  `maintenance_work_mem` only makes sense together with a larger
  `--shm-size` / `shm_size:` on the db container.

**`CONCURRENTLY` semantics, in the owner's terms.** No read or write is
blocked. The build is long, and if it dies — OOM killer, cancel, network
drop, container restart — Postgres does **not** roll it back: it leaves the
index present and marked `indisvalid = false`. That state is silent (it
costs insert time on every sync and serves no query), which is why `0029`
post-validates its own build and **raises** instead of reporting success.

**Recovery.**

```bash
# "did it actually build?" — one row, `t` is the answer
psql "$BOR_DATABASE_URL" -c "select i.indisvalid from pg_index i
  join pg_class c on c.oid = i.indexrelid where c.relname = 'ix_chunks_embedding_hnsw';"

# the recovery for BOTH cases is the same command:
uv run alembic upgrade head
```

* `indisvalid = f` after a killed build → `upgrade head` drops the INVALID
  remnant and rebuilds (the migration's own self-heal).
* `relation "ix_chunks_embedding_hnsw" already exists` → the build committed
  but the version stamp did not land. `0029` handles the INVALID case;
  a VALID leftover of the same name is dropped by hand and re-run:
  `DROP INDEX CONCURRENTLY ix_chunks_embedding_hnsw` then
  `alembic upgrade head`.
* Watching a long build from a second terminal:
  `select * from pg_stat_progress_create_index;`
* Rolling back is cheap and touches no data:
  `uv run alembic downgrade -1` is `DROP INDEX CONCURRENTLY IF EXISTS` — an
  index is a derived structure, and the schema underneath is 0028's exactly.

**After it lands.** Turn on the index in the app's own terms and confirm the
owner-visible parity, in this order:

```bash
uv run python -m scripts.eval_retrieval --from-file tests/fixtures/retrieval_battery.txt
uv run python -m scripts.api_payload_audit          # the app must be running
```

The battery's verdicts should read exactly as they did before the build
(the A7/A8 contract — `top_score` included). The tier's own speed is the
part that changed: **~550 ms → ~12–15 ms** per cosine search at production
order, and the plan flips from `Limit → Sort → Seq Scan on chunks` to
`Limit → Nested Loop → Index Scan on ix_chunks_embedding_hnsw` — verifiable
with `EXPLAIN` on the tier's statement, or by reading
`tests/integration/test_chunks_hnsw_recall.py`'s pin of the same thing.

**Order of operations on a 4 GB-class box.** Land the phase 183–187 memory
work first (`passive_deletes` cascade, the session-release cadence, the
prune's six-column read, the deferred-`content` fast path, the reported
`rss_mb`/`rss_peak_mb` pair — see
[Operational notes: the sync's memory](#operational-notes-the-syncs-memory-phase-187)).
The box that OOM-killed the 2026-10-15 sync is the box this build lands on;
1.6 GB of new index plus a graph build on top of a sync that still holds its
walked documents is how a build gets killed and leaves an INVALID graph for
somebody else to notice.

**Reverse-proxy body size:** a legal chat/save POST is now large — one
conversation can carry a full-length answer (see
[Conversation payload limits](#conversation-payload-limits)) — so an
nginx-style proxy in front of the app needs
`client_max_body_size` **≥ 16 MB** (the app itself sets no body limit);
at the default 1 MB the follow-up of a long answer fails at the proxy
instead of reaching the app.

---

## Troubleshooting

- **`401` from aipi** — set `BOR_LLM_API_KEY` (or `$AIPI_KEY`).
- **`litellm.UnsupportedParamsError … encoding_format`** — the aipi proxy
  (litellm `openai_like`) rejects the `encoding_format` parameter. The app
  already works around this by POSTing a minimal `{model, input}` payload.
  If you see this, you are likely calling the endpoint with a different
  client — drop the parameter (or set `litellm.drop_params = True` on the proxy).
- **Embedding dimension mismatch** — aipi changed models; run
  `uv run python -m scripts.llm_probe`, update `BOR_EMBEDDING_DIM`, then
  drop + recreate the chunks table.
- **No attach button / images ignored on import** — image support is
  off (the default). Set `BOR_IMAGES=1` in `.env` and restart — only
  when `BOR_LLM_CHAT_MODEL` supports vision. Turning the toggle off
  never destroys existing image documents (syncs prune-guard them).
- **An `.svg` shows raw XML instead of the drawing** — image support is
  off, so the file is a plain **text** document (phase 181). Set
  `BOR_IMAGES=1` and re-sync: the next run converts the row in place
  (bytes copied, `image_url` live, no re-import by hand). An `.svg` the
  extractor refuses — over 2 MiB, carrying a `<!DOCTYPE`/`<!ENTITY`
  declaration, or malformed XML — stays a text document and is counted
  in `images_failed`.
- **Honest deflection** (the amber deflection bubble) — every question
  passes the honesty gate: deflection happens
  when the best cosine similarity is below `BOR_RELEVANCE_THRESHOLD`
  (default `0.62`) **and** the lexical signal is absent or not
  vector-corroborated (`fts_hits = 0`, or best cosine below
  `BOR_LEXICAL_SUPPORT_FLOOR` — default `0.35` — A8 revised 2026-09-14).
  A weak cosine with a lexical hit (name-your-tool questions) still gets
  a grounded answer, provided the best cosine clears the floor. When it
  does deflect, the LLM prompt carries weak-hit *titles only* (no document
  content), the reply admits it has no notes on that and offers 2-3
  concrete alternative questions about things it **does** have notes on,
  the bubble renders amber with *"Maybe try"* chips derived from the
  closest indexed titles, and the `query_log` row records
  `deflected=true`. This is a feature, not a bug. With a small local
  model, a rare turn may still answer from general knowledge with an
  explicit disclosure when retrieval was borderline — the gate (A8
  revised) minimizes this by keeping vector-unsupported docs out of the
  grounded prompt, and the disclosure is surfaced, never silent.
- **Answers deflect too often / too rarely** — tune
  `BOR_RELEVANCE_THRESHOLD` (lower = answers more, higher = more honest
  deflection) and `BOR_LEXICAL_SUPPORT_FLOOR` (the best-cosine floor a
  lexical hit must clear to flip HIGH — lower = lexical hits ground more
  easily): `0.0` ⇒ the gate leans entirely on FTS hits; `1.0` ⇒
  everything deflects unless a lexical hit is vector-corroborated. The
  `embed` model's cosines cluster in a ~0.6–0.85 band on the live KB, so
  the default is `0.62`. Check real scores:
  ```sql
  SELECT question, top_score, fts_hits, deflected
  FROM query_log ORDER BY created_at DESC LIMIT 20;
  ```
- **KB offline banner in the chat** — Postgres isn't running:
  `podman compose up -d db`.
- **Stuck "Thinking…"** — the LLM is slow or down; the client's 300 s
  pre-token guard (`TURN_TIMEOUT_MS`) turns a hung stream into the error
  state automatically.

---

## Planning & Architecture

The architecture, LOCKED decisions, and the phase roadmap live in
[`.agents/PLAN.md`](.agents/PLAN.md); per-story specs in
[`.agents/user_stories/`](.agents/user_stories/).