brain-of-reese/README.md

# 🧠 Brain of Reese

A chippy, honest **RAG chatbot** over your homelab documentation — git
repositories, local directories, and uploaded archives registered on the
admin **Sources** page. Ask it anything about your notes — markdown,
YAML, JSON, Python, plain text, and more — and it retrieves the relevant
chunks with **hybrid search** (pgvector cosine ∪ Postgres full-text
search, fused with Reciprocal Rank Fusion), feeds the **whole relevant
document** to a **self-hosted LLM** (`turbo` via
`https://aipi.reeseapps.com/v1`), and streams a grounded answer back.

If it doesn't have notes for your question, it admits it — and offers
2-3 concrete alternative questions about things it **does** have notes on.

Image support is opt-in (`BOR_IMAGES`): standalone images become
retrievable documents, and you can attach an image to a question —
see [Image support](#image-support).

> **Updated your notes?** Re-run the import — it's idempotent and only
> re-embeds what changed:
> ```bash
> uv run python -m scripts.import_docs --prune
> ```
> See [Updating the documents](#updating-the-documents) for details.

---

## Quick Start

```bash
# 1. Dependencies
uv sync

# 2. Configure (copy and edit)
cp .env.example .env
# Set BOR_LLM_API_KEY — your aipi key (falls back to $AIPI_KEY)

# 3. Start the database
podman compose up -d db
podman compose ps          # wait until "healthy"

# 4. Apply migrations & import
uv run alembic upgrade head
uv run python -m scripts.import_docs

# 5. Run the app
uv run uvicorn app.main:app --reload
# → http://localhost:8000  (chat)
```

**Stack:** FastAPI · Pydantic v2 · SQLAlchemy 2 · Alembic · pgvector ·
vanilla HTML/CSS/JS (no CDN) · Playwright E2E

---

## Pages at a glance

The app is a single-page application — all views load from `/` with a
client-side router. Direct URLs deep-link to the matching view.

| Page | URL | Who can see it |
|------|-----|----------------|
| **Chat** | `/` | Everyone (token gate for non-shared) |
| **RAG / Knowledge base** | `/sources.html` | Admin only |
| **Git sources** | `/git-sources.html` | Admin only |
| **Tuning** | `/tuning.html` | Admin only |
| **Saved chats** | `/history.html` | Admin only |
| **Access tokens** | `/tokens.html` | Admin only |
| **Theme** | `/theme.html` | Admin only |
| **Document viewer** | `/document.html?source=X&path=Y` | Token users & admins |
| **Login** | `/login.html` | Everyone |
| **Shared chat** | `/shared/<token>` | Everyone (anonymous) |
| **Edit doc** | `/doc-edit.html` | Admin only |

### Chat (`/`)

Ask questions; answers stream in with **source chips** that cite the exact
documents used (a cited image document gets no chip — its compact
figure is the citation, phase 130). Clicking a chip or figure opens the
document in an **almost-fullscreen modal** on the same page (no new tab);
the browser **Back** button closes the modal — instead of navigating the
view behind it — and Forward re-opens the same document (phase 144). The
modal's **Full page** link opens the dedicated `/document.html` viewer in a
new tab carrying its origin: a document opened from Chat returns through
the viewer's back link to Chat, one opened from Sources returns to Sources
(phase 153, feeding the phase-13 `back` contract on every path). The
onboarding suggestion chips
follow the opening questions of the 3 most recent saved chats (each
chat's first question — a follow-up never chips, phase 103) — on a
fresh deployment they seed from `BOR_SUGGESTIONS`. A long opening
question caps at 90 displayed characters with a `…` (phase 157) — the
full text stays in the hover tooltip, and clicking the chip submits
the full question.

The document's meta row (the chat modal **and** the standalone viewer at
`/document.html`) carries a **Copy path** button: one click copies the
document's `source/path` (the exact form the `read` tool, the chips, and
the Sources table use) to the clipboard; on a non-secure (http) homelab
origin the browser rejects the clipboard, so the click instead offers the
full path in an inline fallback field that selects itself on focus —
click or Tab, then copy (phase 153).

**Edit** the newest question (phase 159): the latest user bubble carries
an **Edit** button that swaps the bubble for an inline textarea
prefilled with the question (the 4,000-character cap is enforced with
`maxlength`; **Ctrl/⌘+Enter** submits, **Esc** / **Cancel** restore the
original byte-identical, and a whitespace-only save is rejected inline).
Saving rewrites the question in the chat and the saved chat in place,
drops every turn that followed it, and re-asks the trimmed question —
a fresh answer streams in the old answer's place (the question is
never duplicated; an attachment on the question survives the edit and
the re-ask stays text-only). A turn that failed with "Question too
long — trim it and re-ask" (or a mistyped question) is fixed by
trimming it in the editor instead of retyping it. Earlier bubbles
never carry the button (last-question-only scope, owner-locked).

**When a long answer spends its research budget** (phase 160): the
answer's meta row carries a **Continue researching** button (row order
Tune → Retry → Continue → Save-as-doc; not admin-gated — chat is
public, so any visitor can keep a long research session going). Each
tool result tells the model how many tool calls remain; calls
attempted after the budget is spent are **rejected** (not executed,
not shown) instead of being dumped into the chat, and the turn ends
with a final no-tools reply — the model is coached to answer from the
results it already has and to tell you it can keep going, and a
deterministic "budget spent" fallback line stands in if that reply
carries no visible content. Clicking the button sends one plain
`continue` turn — the budget is per-turn, so the new turn has a whole
new round budget, and nothing server-side knows or cares that this is
a continuation (A10, stateless). The button is inert while a turn is
streaming and single-shot (a stale button is removed when the new
turn starts); the flag is persisted on the saved record, so the button
restores on a refresh — but a shared chat never shows it.

### RAG / Knowledge base (`/sources.html`)

The indexed document list. The *Path* column opens each document in the same
**almost-fullscreen modal**. Document rows carry a *Created* date and folder
rows an *Updated* date (phase 106) — sourced at sync time (git commit dates
for git sources, restored metadata for archive uploads, file mtime otherwise;
unresolvable or future dates fall back to today) — and the admin can correct a
document's date from the document viewer. The date also feeds the small
recency boost in retrieval ranking (`BOR_RECENCY_BOOST`). Admin-only —
anonymous visitors see a sign-in gate instead.

### Git sources (`/git-sources.html`)

The admin-managed source registry: add or remove git repositories, upload
source archives (`.tar`, `.tar.gz`, `.tgz`, `.zip`), and register local
directories — all in one table with a `kind` discriminator. No `.env`
editing, no restart. The **Sync sources** button clones/pulls/walks every
source and re-imports in one click.

### Tuning (`/tuning.html`)

Set global instructions that are injected into the system prompt of every
chat turn. Add notes like *"be more concise"* or *"assume I'm on NixOS"*.
List, edit, or remove them here — no conversation required.

### Saved chats (`/history.html`)

Every conversation is saved automatically. Click a title to return to that
chat. Admin-only view.

### Access tokens (`/tokens.html`)

Generate tokens and hand them out — a token opens chat, the answers, and
the documents they cite. Shared chats stay open to everyone.

### Shared chat (`/shared/<token>`)

A read-only snapshot of any conversation, shareable via a link. Open to
everyone — no login or token required.

### Edit doc (`/doc-edit.html`)

Review and adjust an AI-generated answer before committing it to the docs
repository. A draft that should never be pushed can be **Discard**ed from
the same page (permanent; admin-only). Admin-only flow page.

---

## Admin & Sign-in

Brain of Reese has exactly **one account: the admin (you)**. Signing in
unlocks the **full Sources catalog**, the **answer-tuning** controls, and
**token management**. **Shared chats** are the only content that stays open
to anonymous visitors.

### Setup (one-time)

```bash
python -c 'import secrets;print(secrets.token_hex(32))'   # → paste into .env
```

```env
BOR_ADMIN_PASSWORD=your-password        # plaintext — homelab scope, by design
BOR_SESSION_SECRET=<the hex from above> # signs the session cookie
```

**Fail-loud:** while either variable is empty the app refuses to start,
naming the missing one(s).

### How authentication works

| Endpoint | Purpose |
|----------|---------|
| `POST /api/login` | Password → signed `bor_session` cookie |
| `POST /api/logout` | Clears session cookie |
| `GET /api/whoami` | `{"authenticated": bool, "role": "admin"|"user"|"anonymous"}` |
| `POST /api/token-auth` | Token (`bor_…`) → same cookie, role `user` |

The cookie uses `same_site="lax"`, `https_only` off — **no HTTPS
enforcement** (homelab HTTP; the cookie is single-admin convenience, not a
cloud boundary). Max age is `BOR_SESSION_MAX_AGE` (default `43200` = 12 h).

### Access tokens

The admin can hand out access without sharing the admin password.

1. **Generate:** sign in → **Tokens** in the navbar. Type a label and hit
   **Generate** — a `bor_` + 32-hex token appears in the *shown once* block.
   Copy it now: only its SHA-256 hash is stored.
2. **Use:** the holder enters the token at the in-app gate. The browser
   caches it in `localStorage["bor.token"]` for silent re-auth on reloads.
3. **Revoke:** click **Revoke** on the token's row. Revocation is immediate.

A token user can chat and open documents — nothing else. The Sources
catalog, git sources, tuning, and history stay admin-only.

---

## Using the App

### Updating the documents

The knowledge base is refreshed by **re-running the import**. It is
idempotent and delta-based: each file's RAW BYTES are hashed (sha256,
streamed) and checked against the last index's stamp *before* any
extraction or vision work — an unchanged file costs one hash + one row
lookup (no re-extract, no vision, no re-embed), while a changed file
re-indexes fully. The one exception is a PDF row that still claims a
page strip: it is re-gated from its current bytes (no vision, no LLM —
see "The strip repairs itself" below). A refresh after a normal editing
session takes seconds — pictorial PDFs included (their expensive
first-index vision burns only once per file version):

```bash
# After editing/adding/removing notes:
uv run python -m scripts.import_docs                 # re-index what changed
uv run python -m scripts.import_docs --prune         # also drop deleted files

# Point it at extra directories (repeatable):
uv run python -m scripts.import_docs --source ~/SomeOtherDocs
```

With **git-based sources** each run first pulls the latest commits of your
repos, so this same command is the whole update loop: *commit → re-run*.

Then check the **Sources** page (`http://localhost:8000/sources.html`):
the *documents* / *chunks* counters and *last indexed* timestamp should
reflect the new files.

- The import prints one line per file and ends with a greppable summary,
  so it is safe to run from cron or after every commit.
- Indexed formats: **md, markdown, txt, yaml, yml, json, py**, the Podman
  quadlet family (**container, network, volume, image, pod, kube, swap,
  os, endpoint**), **j2** Jinja templates — and, **opt-in**, **pdf**
  (add the `pdf` token to `BOR_IMPORT_EXTENSIONS`; see
  [PDF documents](#pdf-documents-phase-126)).
- Hidden files and common cache dirs (`.venv`, `node_modules`, `.git`,
  `__pycache__`, etc.) are skipped — unless the per-source **Index hidden
  folders** checkbox on the Git sources page is on for that source
  (phase 105): then dot-prefixed folders/files with an importable
  extension are indexed like visible ones, and switching it off prunes
  them on the next sync.
- Non-markdown files get format-aware chunking (YAML keys, JSON keys,
  Python classes/defs).
- With `BOR_IMAGES=1`, standalone images (`png, jpg, jpeg, webp, gif,
  bmp`) are indexed as documents too — see
  [Image support](#image-support).
- A run that changed the KB also regenerates the **KB overview** and the
  **folder summaries** (the drill-down catalog's stored per-folder
  descriptions) with the `lite` model — best-effort: a summary failure
  never fails the import. The stored overview ends with a deterministic
  `Sources: <name> (<k> doc|docs), …` line — every source with its exact
  document count, catalog order, computed without a model call — so no
  source (not even a 1-doc one) is invisible to `<knowledge_base>`
  (phase 162); the `LLMError`/zero-doc paths leave the previous row
  (tail included) untouched. Because the sync only rewrites the folders a
  change touches, a one-shot
  `uv run python -m scripts.regen_folder_summaries` refreshes EVERY
  stored folder summary in one run (full mode: every candidate folder
  is regenerated, owner-edited rows are kept, one `lite` call per
  folder, one cron-safe summary line; exit 0, non-zero with a named
  stderr line on a boot/DB failure).

### Git-based sources

Rather than pointing the import at local folders, point it at
**git repositories**. The admin **Git sources page** (`/git-sources.html`)
is the primary management surface — add or remove repositories there and
the list is stored in Postgres.

```env
# .env — the fallback list (fresh setups, or until the admin page
# stores a source; the page becomes the source of truth)
BOR_GIT_SOURCES=https://git.reeseapps.com/reese/homelab.git,git@github.com:reese/deployments.git
BOR_SOURCES_DIR=~/bor-sources   # default; each repo lands in <dir>/<repo-name>/
```

- Public repos need no setup: auth is whatever the machine supplies —
  HTTPS via the OS credential helper, or SSH via your key. The API
  accepts `https://`, `http://`, `ssh://`, `git@`, and `file://` URLs
  (phase 135: a `file://` local-transport repo — e.g. on a mounted
  volume — is a valid source too).
- Private repos: the add/edit form on the Git sources page takes an
  optional **masked token** field (phase 121). The token is stored in a
  dedicated DB column and injected into the clone URL only at
  sync/clone time — it is never returned by any API response or shown
  in the UI (legacy rows that already embed `user:token@` in the stored
  URL keep working — the token is sanitized on output).
- A git source can pin the **branch it mirrors** (phase 135): the
  add/edit form's optional branch field (or the API's `branch`) stores
  the branch in the `git_sources.branch` column — a fresh clone checks
  that branch out, an existing checkout tracks it (`git fetch origin
  <branch>` + `git merge --ff-only FETCH_HEAD`), and a checkout sitting
  on a *different* branch fails the sync loudly with a remove-and-
  re-add message (never a silent switch). Rows without a branch —
  every pre-existing row and the env-fallback rows — track the
  checkout's default branch (byte-identical clone behavior).
- Every run **clones** (first time, full history) or brings an existing
  checkout to the **head of the tracked branch** (phase 136: `git fetch
  origin <branch>` + `git merge --ff-only FETCH_HEAD`) and then indexes
  the checkout. If the remote was **force-pushed** — a fast-forward is
  impossible — the checkout is reset hard to that head with one loud
  log line **instead of failing the sync**: the checkout is a cache of
  the remote head, and the KB is rebuilt from it on the same run, so a
  force push is a logged, self-healing event. A genuinely broken fetch
  (dead network, a branch deleted upstream) still fails the sync
  loudly. A legacy shallow checkout is unshallowed once on its first
  post-phase-107 sync (`git fetch --unshallow` — the one-time
  self-heal) so commit dates stay real.
- A failed sync aborts the run — no partial junk.

### Archive upload sources

Upload a `.tar`, `.tar.gz`, `.tgz`, or `.zip` archive to make it a source.
The form on the Git sources page — and the `POST /api/git-sources/upload`
route behind it — accepts archives and unpacks + registers them (no scan —
the upload indexes nothing; the **Sync sources** button on the RAG page
performs the scan, which leaves room to edit the new source's ignore list
first).

- Re-uploading the same filename replaces the source **in place** (no
  second folder, no duplicate row).
- After the 202 the form settles into the ready-for-sync line
  "Uploaded <source> (<stamp>) — press Sync sources to import it." —
  the completion stamp (phase 154: the same date-qualified freshness
  rule as the sync badge — `HH:MM` on the same local day,
  `M/D, h:MM AM/PM` otherwise, `M/D/YYYY` across years) rides the
  line on EVERY render path, so a line that outlives its upload is
  self-dating: after a reload the source name is gone (it was
  page-local) but the stamp survives, coming from the server
  payload's `finished_at`.
- Zip-bomb guard: absolute member paths, `..` traversal, and symlinks
  escaping the unpack folder are rejected.
- Archives unpack under `BOR_UPLOAD_DIR` (default `~/bor-sources/uploads`).
- `BOR_UPLOAD_MAX_MB` (default 512) caps both compressed and extracted size.

### Local directory sources

A plain directory can be a first-class source too. It shares the git
sources' **one table** (the `git_sources` registry with a `kind`
discriminator: `git` | `local`), **one admin page**, and **one Sync button**.

- Register via API: `POST /api/git-sources` with `{"kind": "local", "path": …}`.
- Sync walks it directly — no clone, no checkout copy.
- The directory is re-verified to exist at sync time; a missing directory
  fails the run loudly.

### Sync from the UI

The **Sync sources** button on the **Sources** page (admin only) runs the
whole git-source refresh in one click:

1. **clone/pull + walk** every configured source — git repos and local
   directories, including uploaded archives.
2. **re-import with prune** — files deleted upstream leave the index.
3. **regenerate the KB overview** (the `<knowledge_base>` outline every
   chat turn injects) — only when the import changed the knowledge base.
4. **refresh the folder summaries** (the drill-down catalog's stored
   per-folder descriptions) — a changed knowledge base rewrites ONLY the
   folders the changed files touch (each file's folder chain up to its
   source root — phase 127: one changed file re-stamps a handful of
   rows, not the whole table); an unchanged re-sync instead gap-fills
   any missing rows (`only_missing`).

- States: while polling `GET /api/sync/status` every 2 s, the button
  becomes the spinner (icon only — its label parts are hidden while a
  run is in flight, and its accessible name is the full live label),
  and the full live detail — "Syncing… <file> (n/m)" during the import
  phase (the live file, its `(n/m)` file position, and — while a file's
  embed batch is in flight — the pinned `· chunk n/m` embed position,
  batch granularity: the first chunk of the in-flight batch over the
  file's total content chunks; and — while a PDF's images are being
  described — the `· image n/m` vision position: the 1-based image
  about to be described over the file's total images, ONE continuous
  counter across the PDF's three vision spans, so a long image pass
  visibly climbs instead of sitting frozen on the last chunk
  position) — shows in a text box under the button row,
  wrapping (never clipped); the same untruncated label rides the
  button title and the box, which is the SOLE `aria-live` region for
  the sync area (`role="status"`, polite — phase 150 folded the
  pre-phase result-line announcer into the box). At settle the box
  STAYS visible to carry the result line — the counts line
  ("N added · …") after a success, "Sync cancelled" after a cancel —
  and the button leaves the running state (the "Synced <stamp>" pill
  after a success — the date-qualified freshness stamp (phase 154):
  `HH:MM` on the same local day, `M/D, h:MM AM/PM` otherwise
  (`M/D/YYYY` across years — a badge that tells the owner *when*),
  the bare "Synced" when the stamp is missing — the retry-ready
  "Sync sources" pill after a failure); the box hides only when its
  text is empty (a failed sync
  — the banner + modal carry the error — and the true idle). No
  client-side timeout — a clone + embed can legitimately take minutes.
- One sync at a time: a second trigger while a run is in flight gets a
  `409`.
- **Cancel**: while a run is in flight, a **Cancel** button appears
  next to the sync button; clicking it posts `POST /api/sync/cancel`
  (admin only) and registers a cooperative cancel (`202`) — the run
  stops at the next pipeline boundary (a git clone/pull already in
  flight runs to completion) and settles into a neutral `cancelled`
  terminal state (`error: null` — intent, not failure: a "Sync
  cancelled" result line, no error banner or modal, the button
  retry-ready) that the status endpoint reports; with no run in
  flight the request gets a `409` (the UI treats it as a harmless
  race — the poll settles). A sync started after a cancel runs clean
  (the importer's content-hash delta re-imports what the cancel
  skipped).

---

## Thinking

The self-hosted `turbo` model reasons before it answers. That reasoning is
streamed with the turn as `thinking` SSE events and shown in **
collapsible "Thinking" blocks** — one block per
contiguous thinking run, in arrival order (phase 158) — all wrapped in
**one collapsible "Reasoning" bubble** above the answer bubble
(phase 161): the turn renders as a chronological timeline
(Thinking → Tool call (1) → Thinking → … → answer), condensed into a
single bounded live window (320 px, one scroll surface) that follows
the tail while you watch.

The bubble is created on the turn's first thinking (or tool) frame and
stays **open** from creation until the turn settles — including a
mid-round reply (the model answered, called a tool, then thinks again:
the reply streams into the answer while the bubble stays open, and the
new thinking run gets its own block inside it instead of resuming on
top of it). It folds when the turn ends, is stopped, or fails. While
open, its summary carries a live status ("Thinking…" /
"Calling <name>…") and reads "Reasoning — N steps" once the turn
settles; a turn with no thinking and no tools renders no bubble at all
(the answer layout is byte-identical to pre-phase).

The per-run blocks are plain and always visible inside the open
bubble: each block appears and fills in live while its run streams,
and the runs simply stop accumulating frames when the first answer
token lands, when the turn ends, or when the turn is
stopped/failed. They are distinct from each other by label + rail:
the brand-ink "Thinking" label on a brand rail, the accent-ink
"Tool call(s) (N)" label on a neutral rail.

The tool calls ride the same timeline: each contiguous run of tool
calls gets its own "Tool call(s) (N)" block (one per run, in
arrival order; always visible while the bubble is open), and the
per-line elapsed-seconds suffix targets the newest line of the
active run.

The timeline is persisted with the saved turn (an optional `segments`
array — the per-run order; the concatenated `thinking` and the `tools`
record stay for history and older clients), so a reload and the shared
page re-render the same timeline **collapsed inside the Reasoning
bubble**, with the step-count summary.

To hide it, set `BOR_STREAM_THINKING=0`.

---

## Agent document tools (ls + read + grep)

Retrieval only puts the top documents in context. When an answer depends on
a file a note *references* ("the exact JSON shape is in
`example-record-file.json`"), the model can extend its own context with
server-side tools — on **grounded** (high-relevance) turns only:

| Tool | Description |
|------|-------------|
| **`ls`** | Lists the knowledge base as a **drill-down tree**, one level at a time: no `path` = every source (document count + stored description), a source name = that source's top level, a `source/folder` path = one level deeper; file lines are `source: X \| path: Y \| title: Z` |
| **`read(path, offset?, limit?)`** | Appends one indexed document's text to context by its combined `source/path`; optional `offset` (1-based first line) + `limit` (line count) read just that part of a large document; very large reads are truncated at `BOR_READ_MAX_CHARS` with a notice that teaches the workflow — grep reports line numbers, then read that part with `offset` + `limit` |
| **`grep(pattern, path?)`** | Searches indexed documents for an exact string (case- and accent-insensitive fixed substring — accents fold: `besta` matches `BESTÅ`, phase 162); returns up to 20 `source/path:line: text` matches; an optional `path` limits to one document |
| **`show_image(path)`** | *(only when `BOR_ANSWER_IMAGES=1`)* shows one standalone image document as part of the answer — image documents only, at most one per answer |

(See [Answer images](#answer-images-phase-124) for the `show_image` tool.)

**The prompt teaches the search discipline (phase 162).** A summary or an
outline is a hint, never an inventory: the system prompt tells the model to run
`ls` with no `path` — the full source list — before ever claiming the knowledge
base lacks something, and to `grep` the artifact's part number or a filename
token. One grep pattern is one string, so a name's split tokens (`frame`,
`white`) and its part number (`AA-1272068`) each need their own grep; and a
note saying a file is gitignored describes its **repo** status, not whether it
is indexed as a source.

**Git-source provenance (phase 145).** When a document comes from a git
source, its `read` result also carries an `origin:` line (the repository's
URL, credentials stripped) and, when the source pins one, a `branch:` line.
The prompt teaches the model to include that original source in its answer —
a link to the repository with the document's path named (and the pinned
branch when one is shown) — so you can open the complete original in one
step rather than asking again for the source. Local (non-git) sources carry
no `origin:` line, so nothing is added and no link is invented. The same
origin rides the document-content payload (`origin_url` + `origin_branch`
when pinned), and the document viewer — the chat's doc modal AND the full
`/document.html` page, one render site — shows it as an **Original**
external link in the document's meta row (the full repository URL as the
link text, opening in a new tab; the pinned branch rides the link's
title). Non-git documents show nothing new.

Each call is executed against Postgres only (no extra LLM round trip) and
streamed as an SSE `tool` frame. In the chat, each call shows a transient
**calling-tool status** alongside "thinking" (the send button keeps its
busy state — "Stop" — for the whole turn).

**The search line shows where it searches (phase 146).** When a `grep`
re-searches one document (the optional `path`), the chat's tool line and
the calling-tool status show the document the search runs in —
`🔎 Searching for <pattern> in <source/path>` — so a whole-KB search and
a re-search of a single document read as distinct searches instead of
looking like the model is repeating itself. The scope rides the SSE
`tool` frame additively (`path`, present on scoped greps only) and is
persisted on the saved turn's tool record, so it re-renders after a
save → restore; a bare `grep` renders exactly as before (frame, line,
status, and saved shape all byte-identical to pre-phase — old saved
turns restore unchanged).

The model may call tools as many times as needed, bounded by a round cap:

| Env | Default | Meaning |
|-----|---------|---------|
| `BOR_AGENT_MAX_ROUNDS` | `10` | hard cap on agent tool rounds per grounded turn (0 = no tools, the kill switch); every tool result carries the model's remaining count (phase 160) |

At the cap, further tool calls are **rejected** (never executed, never
shown) and the turn ends with a final no-tools reply: the model is
coached to answer from the results it already has and to tell the user
they can say "continue" to keep researching; a reply with no visible
content answers with a deterministic "budget spent" fallback line
instead of an error frame (phase 160, the recorded revision of the
phase-71 cap-path terminal). The `done` frame carries an additive
`budget_exhausted` key (ABSENT on every non-cap turn) that drives the
**Continue researching** button on the answer's meta row — see the
[Chat (`/`)](#chat--) section.

---

## Tuning your answers

*Admin-only* — sign in first. If an answer isn't quite right — too chatty,
wrong assumption, missing context — **tune** Brain right there:

1. Press **"Tune"** in the meta row under any completed answer (deflected
   ones included).
2. Type a short instruction (1–2000 chars), e.g. *"be more concise"* or
   *"assume I'm on NixOS"*, and **Save**.

The note is stored in Postgres (`steering_notes`) and read into the
**system prompt of every subsequent chat turn** as a `<tuning>` section
(numbered, oldest first, capped at `BOR_STEERING_MAX_CHARS` chars — default
8000, overflow marked `[…truncated…]`).

List or remove notes from the **"Tuning"** button in the chat header (count
badge, newest-first, per-note delete). The API is stateless JSON if you
prefer curl:

```bash
curl -s localhost:8000/api/steering                                  # list
curl -s -X POST localhost:8000/api/steering \
     -H 'Content-Type: application/json' -d '{"note": "be more concise"}'
curl -s -X DELETE localhost:8000/api/steering/<note-id>              # remove
```

---

## How retrieval works (hybrid)

Every question is embedded and also lexically tokenized (OR-joined, English
stemming) and searched **twice** against Postgres:

1. **Vector** — pgvector cosine top-N (default `BOR_HYBRID_VECTOR_CANDIDATES=100`)
2. **Lexical** — a stored `tsvector` (GIN-indexed) matched with `to_tsquery`,
   top-N by `ts_rank` (default `BOR_HYBRID_LEXICAL_CANDIDATES=30`), led by
   **name hits** — documents whose *path* matches a question token (`gitea`
   → `gitea/…`, `qwen38` → `qwen3.8-27b-…`; phase 119). Name normalization
   is **accent-folded** (phase 162): a question's `BESTÅ` folds to `besta`
   and matches the ASCII filename, and an accented filename (`BÅsta.md`)
   matches an ASCII question — both directions, stdlib fold only.

The two ranked lists are fused with **Reciprocal Rank Fusion**
(`score = Σ 1/(k + rank)`, `BOR_RRF_K=60`) — a chunk in both lists scores
nearly double, which lets a name-your-tool question ("gitlab") find its own
document even when the question embeds close to generic templates.

The **honesty gate** (A8) then answers (HIGH) when the best cosine is ≥
`BOR_RELEVANCE_THRESHOLD` (default `0.62`) **or** when at least one chunk
matched lexically (`fts_hits > 0`) **and** the best cosine clears
`BOR_LEXICAL_SUPPORT_FLOOR` (default `0.35`) — the vector signal must
corroborate a lexical hit (A8 revised 2026-09-14). It deflects (LOW)
otherwise, including a weak single-token hit with vector-unsupported
docs. The top `BOR_TOP_N_DOCS` full documents are what the LLM sees.

---

## Document summaries

Raw yaml/json/py/txt embeds badly — flags and keys are not language, so
retrieval can miss exactly the documents that are all configuration. At
import time, **every** new or changed document gets a `lite`-model summary —
**markdown included**: the summary is a natural-language retrieval target
and the seed text for the onboarding suggestion blocks (phase 118 retired
the old "markdown is already natural language" exclusion):

* The summary is stored on `documents.summary` **and** indexed as one extra
  embedded chunk (`chunks.is_summary`, position −1), so hybrid search has a
  natural-language target to hit instead of the raw text.
* The last line is a **code-deterministic** pointer — `Source: <source>/<path>`
  — appended by the app, never model-generated.
* The model only sees the first `BOR_SUMMARY_MAX_CHARS` (default 12000)
  characters; overflow is cut and marked with `[…truncated…]`.

Summary generation is best-effort: if `lite` fails, the document is still
indexed (without a summary), the failure is logged, and counted in the
import summary line (`summaries=N summary_errors=N`).

---

## Image support

One toggle gates both image capabilities: `BOR_IMAGES` (default
**off**). Enable it **only when your chat model
(`BOR_LLM_CHAT_MODEL`) supports vision** — the chat model is the vision
model: it describes image documents and answers questions with attached
images. The embedding model never sees pixels in either flow.

### Image documents

With `BOR_IMAGES=1`, a standalone image file — inside an uploaded
source archive, or as a file in a git/local source walk — becomes a
first-class document:

- **Described by the chat model.** On first index, a single multimodal
  chat-model call describes the image. The description becomes the
  document's `content` **and** `summary` — and is the *only* text that
  gets embedded, so hybrid search has a natural-language target to hit
  instead of pixels.
- **The bytes persist.** Each ingested image is copied to
  `BOR_IMAGE_DIR/<uuid>.<ext>` and served from that copy (uploads are
  replaced and checkouts re-cloned, so the served image outlives its
  source file).
- **Retrieved & displayed.** Image docs appear in the Sources catalog
  with a small lazy-loaded thumbnail (progressive enhancement — a fetch
  failure falls back to the doc icon), the document viewer renders the
  image with the description below it, and a cited image doc renders
  **in the chat's sources images row** (phase 130: its own
  `.msg-meta.msg-images` cluster below the chips row — no chip for an
  image doc, the figure is the citation; alt text = the description;
  a failed image load replaces the figure in place with the plain chip,
  so the citation survives a lost byte copy). The agent's `read` tool
  returns the description with a marker telling the model it is reading
  an image document.
- **Prune guard.** While `BOR_IMAGES` is **off**, syncs never prune
  existing image documents — an image doc is invisible to the walk, not
  a deleted file. With the toggle on, a deleted image file prunes its
  doc and its copy.
- **Fail-soft.** A failed or empty description skips that image (no
  row, no copy), logs a warning, and counts in the import summary line
  (`images_failed=N`) — the sync continues.

Accepted formats are `png, jpg, jpeg, webp, gif, bmp`
(`BOR_IMAGE_EXTENSIONS`, case-insensitive) — a separate set from
`BOR_IMPORT_EXTENSIONS`; images are never added via that variable, the
toggle is the single knob.

### Question images

With the toggle on, the composer gains a **paperclip** button. Attach
one image to a question and the chat model answers with text **and**
image:

- **Server-stored, path-persisted.** On send the file is first uploaded
  to `POST /api/chat-images` (capped at `BOR_CHAT_IMAGE_MAX_MB`,
  default 10 MiB — over is a 413) and stored under `BOR_CHAT_IMAGE_DIR`.
  Saved and shared chats carry the returned *path* — never base64.
- **Visible while uploading.** While the upload is in flight the
  preview strip shows a spinner + *"Uploading image…"* (its remove ✕
  disabled) and the Send button is disabled with the *"Uploading…"*
  label — a second press is dead, visually and functionally (no second
  upload, no second turn). On success the state clears and the turn
  proceeds; on failure it clears, the banner shows, and the typed text
  + the attachment stay for the retry (phase 132).
- **Current turn only.** The model receives a multimodal message for
  this question; prior turns' images are never replayed into history
  (the text of a prior turn stands alone).
- **Rendered everywhere.** The image renders in the user bubble, after
  a page reload, and on the shared chat page; a failed load degrades to
  a small "image unavailable" note (never a broken-image icon).
- **Blocked on failure.** If the image upload fails, the send is
  blocked with a banner — the question is never sent without the image
  the user attached.
- **Toggle off** ⇒ the attach control is absent, and an API request
  carrying an image gets an error frame with the hint: *Enable
  `BOR_IMAGES` in the server's `.env` (and restart) to ask with an
  image.*

A question image is **never indexed as a document** — it is turn-local
storage, not a source.

### Answer images (phase 124)

A second, independent toggle — `BOR_ANSWER_IMAGES` (default **off**): when
on, the grounded chat model may show a standalone image document as part
of its answer:

- **The model decides.** The agent is offered a fourth tool,
  `show_image(path)`, and calls it only when the image itself contains
  the answer (a diagram, screenshot, or photo of the thing asked about) —
  never for mere topical relevance. At most **one** image per answer
  (enforced server-side); the text answer is always given in full — the
  image supplements, never replaces.
- **Rendered as an answer figure.** A shown image appears as a
  first-class figure below the answer text and above the source
  clusters (the chips row, then the images row — phase 130) — on the
  live turn, after a page reload, and on the shared chat page; a failed
  load collapses the figure (never a broken-image icon).
- **Image citations on zero-read turns.** A retrieved image document in
  the suggested tier is auto-cited: its figure (in the images row — no
  chip, phase 130) appears even when the model never `read`s it — the
  model's seed block already carries the image's full description, so a
  real model has no reason to read it (this applies with `BOR_IMAGES=1`,
  any `BOR_ANSWER_IMAGES`).
- **Off = byte-identical.** No fourth tool offered, no figure, no
  `answer_image` key in any SSE frame or saved record.

---

## PDF documents (phase 126)

PDFs are **opt-in**: add the `pdf` token to `BOR_IMPORT_EXTENSIONS`
(a stock `.env` never indexes them). Once admitted, a PDF is a
first-class document through every entry point (git/local walks and
uploaded archives):

- **Text, page-aware.** Text is extracted page by page (`pymupdf`); one
  document per PDF, its text-bearing pages joined with `---PAGE-N---`
  separators. Chunking is page-aware — every chunk keeps a `Page N:`
  anchor, so a retrieval hit stays readable in context.
- **Summarized + dated like any document.** The `lite` model summarizes
  it, sync sources its date (git commit / file metadata), and it appears
  in the catalog, the KB tree, and the agent's `ls` / `read` / `grep`
  tools.
- **Embedded images join the image pipeline.** With `BOR_IMAGES=1`,
  images embedded in the PDF are extracted and run through the same
  image-document flow as standalone images (chat-model description,
  persistent copy under `BOR_IMAGE_DIR`, retrievable + displayable);
  with the toggle off the pass is skipped and the PDF text indexes
  normally. Embedded images in a format outside `BOR_IMAGE_EXTENSIONS`
  are skipped with a warning. The extraction is de-duplicated and
  viewer-faithful (phase 151): repeated placements of the same image
  yield ONE document (first placement wins), an image with a soft
  mask (`/SMask`) is stored as the composited image a PDF viewer
  shows (alpha PNG — not the blank base stream), and near-white,
  nearly-uniform images (the table-shading "blank white" underlays)
  are skipped entirely — no row, no copy, no vision call. The run
  counts the drops log-only on the import summary line
  (`pdf_images_deduped=N` `pdf_images_blank_skipped=N`).
- **Document context (phase 137).** Every embedded image is described
  *with* the document it came from: the vision prompt — and the stored
  `image_context` + enriched content — carry an
  `Extracted from <source>/<pdf> (page N):` origin line (always
  present, even for a text-less page) plus the page's own text
  (capped at 6 000 chars; dropped when the phase-133 gate flags the
  text layer as garbage — origin line only). Markdown `![alt](path)`
  references to the image's synthetic path ride along after it
  (phase 125, document block first), so the description names the
  document and the page the image illustrates. Image docs a
  pre-phase-137 sync left without the block (`image_context` NULL or
  markdown-only) are **backfilled on the next sync** — re-described
  once with the context (the parent's page text is re-read from the
  source while the PDF is still there; parent gone → origin line
  only), idempotent (the following sync burns zero vision calls for
  them), fail-soft (a failure leaves the row untouched and retries
  next sync), counted log-only on the import summary line
  (`image_context_backfilled=N`). The stored context is also VISIBLE
  in the chat suggestion block: an image doc's `<document>` body leads
  with a one-line locator — `From <source>/<pdf> (page N)` for a
  PDF-embedded image (with `; Referenced in …` appended when markdown
  reference blocks are stored too — document block first),
  `Referenced in <source>/<path>: "alt"` for a markdown-referenced
  image (phase 125) — the page-text body is never rendered.
- **Garbage text layer → page-screenshot fallback (phase 133).** Before
  indexing, the extracted text passes a deterministic garbage gate
  (empty text layer, or ≥ 30% tag-looking lines, or ≥ 5% non-printable
  chars — the class of PDF that reads as raw XML/binary soup). On
  failure every page is rendered to a screenshot and read by the vision
  model; the stored content is the `---PAGE-N---`-anchored page
  descriptions (a failed page gets a `page N — description unavailable`
  placeholder — fail-soft, the doc still lands). Clean text layers
  never burn a fallback vision call. The run counts the affected PDFs
  on the import summary line (`pdfs_screenshotted=N`). A garbage-text
  PDF takes this fallback instead of the pictorial pass below — the
  precedence is locked, never double vision.
- **Pictorial pages get vision-described (phase 140, on by default).**
  For a PDF whose text layer is usable but thin — the IKEA
  assembly-instruction class (20 of 24 pages: 4–73 chars of markers
  around vector-drawn pictograms) — every page whose STRIPPED text is
  under `BOR_PDF_PICTORIAL_MIN_CHARS`
  characters (default `100`; a deterministic per-page threshold, no LLM
  in the gate) is rendered to a PNG and described by the chat model,
  and the description appends to that page's content as
  `[Page N illustration: …]` — so the embeddings carry what the thin
  text layer couldn't, and a question about the illustrations hits.
  NO page cap: every candidate page is described (A21 — the LLM calls
  are free; a slower sync is acceptable). A PDF with zero candidate
  pages burns no render or vision call — its indexed content is
  byte-identical to pre-phase-140. The run counts the affected PDFs and
  pages on the import summary line (`pdfs_pictorial=N`
  `pdf_pictorial_pages=N`). `BOR_PDF_PICTORIAL=0` turns the whole pass
  off (byte-identical pre-phase-140 content).
- **Served as text — or as pages (phase 148).** The document viewer
  renders the extracted text — never the binary PDF. The exception is
  the two page-qualifying classes: a PDF whose text layer fired the
  phase-133 screenshot fallback, or a PDF whose phase-140 pictorial
  pass described ≥ 1 sparse page. For those, every page is rendered
  to a persistent PNG at index time (under `BOR_PDF_PAGE_DIR`) and
  the viewer — the doc modal AND `/document.html`, one shared core —
  shows the PAGES: a vertical strip of lazy page images with
  "Page N" captions, in place of the text content and the Summary
  panel (the per-page descriptions stay in the DB for Q&A — the
  embeddings and summary are untouched, just not displayed). Since
  phase 163 this strip-only shape is the PRIMARY class only — the
  pictorial class keeps its text behind a Text/Pages toggle, per the
  next bullet. A lost
  copy shows a "Page N unavailable" note for that page only. Text-
  heavy PDFs (usable layer, zero pictorial candidates — and the whole
  sparse class while `BOR_PDF_PICTORIAL=0`) keep the text view,
  byte-identical to pre-phase.
- **Which kind of strip it is (phase 163).** The two qualifying classes
  are not the same problem: the screenshot class has no usable text to
  show, while a book with one sparse page (an illustration map page in a
  36-page text) does. The document row therefore records the strip's
  CLASS beside the page count (`documents.page_strip_mode` —
  `primary` = the screenshots ARE the content, so the strip is the view;
  `additive` = a usable text layer, so the text + Summary view stays and
  the pages sit behind a Text/Pages toggle), and the document-content
  payload carries it as `page_strip_mode`. The key is ABSENT — never
  `null` — when there is no strip, which is also how a row written before
  this phase serializes; the viewer treats an absent class as `primary`,
  so every existing document renders exactly as before until the next
  sync stamps it (the sync's self-heal pass, phase 163).
- **The strip repairs itself (phase 163).** An unchanged PDF is not
  assumed to still have its page images on disk. On every sync, a row
  that claims a strip is re-gated deterministically from the current
  file bytes and the current settings — the same gate the changed path
  uses (`BOR_PDF_PICTORIAL` and `BOR_PDF_PICTORIAL_MIN_CHARS`
  included), with **zero vision or LLM calls** (rendering is CPU-only):
  the class is re-stamped, any `<doc-id>-p{N}.png` missing from
  `BOR_PDF_PAGE_DIR` is re-rendered (write-before-delete, atomically per
  file), and a strip that no longer qualifies — the kill switch turned
  off, the sparse-page bar lowered past it — is cleared along with its
  copies. So a wiped pages directory, or a copy set lost to an
  interrupted sync, repairs itself on the next sync instead of leaving
  "Page N unavailable" forever. It logs one `PDF strip re-gate …` line
  only when it actually acts, and a failure there never fails the sync.
  PDFs without a strip pay nothing extra (no extra read at all).

---

## Caching & deploys

A deploy is a commit — and the browser must see it **without a hard refresh**.
One Starlette middleware (`app/core/caching.py`) applies the rule at the
transport layer:

- **HTML pages** ship `Cache-Control: no-cache`, **no `etag`**, **no
  `last-modified`** — each visit always gets a fresh 200 body.
- **Assets are versioned and cached for a year.** Pages reference their
  CSS/JS with a token (`/assets/styles.css?v=<token>`), and every
  `/assets/*` response ships `Cache-Control: public, max-age=31536000, immutable`.
- **The token is the deploy.** In a git checkout it is the short SHA of
  `HEAD` — so every commit/deploy flips the token and the versioned asset
  URLs change with it. A checkout without `.git` falls back to a stable
  content hash of the `frontend/` tree.

No CDN, no new services, no build-step change: the middleware rewrites the
asset references of the known pages in flight.

> **Deploy note:** the very first deploy onto this scheme needs one normal
> page visit, so the browser revalidates the HTML once and starts requesting
> the versioned assets; every commit after that is picked up automatically.

---

## Configuration reference

Everything is an environment variable (the `BOR_` prefix, plus
`DEBUGPY*`) read from `.env`. **Every variable in
[`.env.example`](.env.example) is documented below** — copy that file
to `.env` and adjust.

### App & branding

| Env | Default | Meaning |
|-----|---------|---------|
| `BOR_APP_NAME` | `Brain of Reese` | Display name everywhere (page titles, header brand, status labels) |
| `BOR_INPUT_PLACEHOLDER` | `Ask me anything…` | Chat composer placeholder |
| `BOR_FOOTER_TEXT` | `Powered by self-hosted models` | Footer line on every page |
| `BOR_ENVIRONMENT` | `development` | `development` or `production` (the prod compose profile sets `production`) |
| `BOR_LOG_LEVEL` | `INFO` | App log level |
| `BOR_STATIC_DIR` | `frontend` | Where static files are served from (the container image sets `/app/static`) |

### Database & concurrency

| Env | Default | Meaning |
|-----|---------|---------|
| `BOR_DATABASE_URL` | local compose URL | SQLAlchemy URL (psycopg) |
| `BOR_DB_POOL_SIZE` | `5` | DB connection pool size |
| `BOR_DB_POOL_MAX_OVERFLOW` | `10` | Overflow connections above `BOR_DB_POOL_SIZE` |
| `BOR_DB_POOL_RECYCLE` | `3600` | Recycle pool connections after N seconds |
| `BOR_CHAT_MAX_CONCURRENT` | `10` | Max concurrent `/api/chat` turns |

### LLM (aipi)

| Env | Default | Meaning |
|-----|---------|---------|
| `BOR_LLM_BASE_URL` | `https://aipi.reeseapps.com/v1` | OpenAI-compatible endpoint |
| `BOR_LLM_API_KEY` | — (falls back to `$AIPI_KEY`) | aipi API key |
| `BOR_LLM_CHAT_MODEL` | `turbo` | Chat model — must support vision when `BOR_IMAGES=1` |
| `BOR_LLM_EMBED_MODEL` | `embed` | Embedding model |
| `BOR_LLM_SUMMARY_MODEL` | `lite` | One-shot completions: document summaries at import, KB overview, folder summaries |
| `BOR_EMBEDDING_DIM` | `768` | Vector dimension (fixed at table creation) |
| `BOR_LLM_RETRIES` | `3` | Retries: chat-turn stream before the first token + empty one-shot summary replies; `0` = off |
| `BOR_LLM_RETRY_DELAY` | `5` | Seconds between LLM retries |
| `BOR_LLM_TIMEOUT` | `120` | HTTP timeout (s) for LLM API calls |
| `BOR_STREAM_THINKING` | `1` | Stream the model's thinking as `thinking` SSE events (`0` = suppress) |

### RAG & prompt budgets

| Env | Default | Meaning |
|-----|---------|---------|
| `BOR_TOP_N_DOCS` | `2` | Ceiling for the cited tier, not a quota (phase 113) — a single strong doc yields one citation |
| `BOR_RELEVANCE_THRESHOLD` | `0.62` | Answer when best cosine ≥ this **or** a lexical hit corroborated by cosine ≥ `BOR_LEXICAL_SUPPORT_FLOOR`; otherwise honest deflection |
| `BOR_LEXICAL_SUPPORT_FLOOR` | `0.35` | Best-cosine floor an FTS hit must clear to flip the gate HIGH (A8 revised 2026-09-14) |
| `BOR_SOURCE_USEFULNESS_FLOOR` | `0.35` | Cosine bar for a citation slot (phase 113): a doc earns the cited tier only when its best-chunk cosine clears it; `0` = no bar |
| `BOR_RELATED_MAX_DOCS` | `2` | Cap on the related-doc tier (the de-emphasized "nearby docs" row — never a citation chip); `0` = no related docs |
| `BOR_SUGGESTED_DOCS` | `5` | Cap on the "start here" suggestion tier (phase 118): top-N doc summaries seeded into the grounded prompt; no cosine floor; ≥ 1 |
| `BOR_SUGGESTION_PREVIEW_CHARS` | `400` | NULL-summary fallback preview in a suggestion block (phase 118): first N content chars + truncation marker; > 0 |
| `BOR_HISTORY_MAX_TURNS` | `40` | Prior client-provided history turns kept per chat turn (phase 74, newest first); `0` = no history (the pre-phase-74 two-message requests) |
| `BOR_HISTORY_MAX_CHARS` | `24000` | Total char budget for the kept history — `len(text) + len(thinking)` per turn (phase 74); a turn that would overflow is dropped whole, never cut mid-answer; `0` = no history |
| `BOR_MAX_OUTPUT_TOKENS` | `32768` | Max answer length in tokens (answers must not be cut off) |
| `BOR_STEERING_MAX_CHARS` | `8000` | Char budget for the `<tuning>` (steering notes) prompt section |
| `BOR_SUMMARY_MAX_CHARS` | `12000` | Cap on document content sent to the `lite` summary model |
| `BOR_KB_OVERVIEW_MAX_CHARS` | `4000` | Char budget for the `<knowledge_base>` prompt section — the stored outline **plus** its deterministic `Sources: …` tail (phase 162); the cap keeps the longest-fitting prefix, so a value below the default clips the tail first (the default always holds both) |
| `BOR_OVERVIEW_INPUT_MAX_CHARS` | `40000` | Cap on the document list sent to `lite` for KB overview generation |
| `BOR_FOLDER_SUMMARY_INPUT_MAX_CHARS` | `8000` | Cap on a folder's document list sent to `lite` for folder summaries (phase 94) |

### Chunking & embedding

| Env | Default | Meaning |
|-----|---------|---------|
| `BOR_CHUNK_TARGET_CHARS` | `2000` | Target chunk size (chars) |
| `BOR_CHUNK_OVERLAP_CHARS` | `200` | Overlap between adjacent chunks (chars) |
| `BOR_EMBED_BATCH_SIZE` | `16` | Chunks per embedding API call |
| `BOR_EMBED_QUESTION_MAX_CHARS` | `1200` | Chat question-embed prefix cap (phase 114): the embed step embeds at most this many chars of the question — the full question still reaches the LLM prompt |

### Hybrid retrieval

| Env | Default | Meaning |
|-----|---------|---------|
| `BOR_HYBRID_VECTOR_CANDIDATES` | `100` | Cosine list width for the RRF fusion |
| `BOR_HYBRID_LEXICAL_CANDIDATES` | `30` | FTS list width for the RRF fusion |
| `BOR_RRF_K` | `60` | RRF damping constant (`1/(k + rank)`) |
| `BOR_RECENCY_BOOST` | `0.0007` | Max additive fused-score bonus a zero-age doc gets (phase 106); `0` = off |
| `BOR_RECENCY_HALF_LIFE_DAYS` | `365` | Recency decay timescale, days (phase 106) — the boost decays by e^-1 (~0.37x) per timescale of age |
| `BOR_NAME_HIT_BONUS` | `0.005` | Selection-time bonus a name-hit doc's best fused score gets (phase 119); `0` = off |

### Agent tools

| Env | Default | Meaning |
|-----|---------|---------|
| `BOR_AGENT_MAX_ROUNDS` | `10` | Hard cap on agent tool rounds per grounded turn (0 = no tools); every tool result carries the model's remaining count (phase 160) |
| `BOR_READ_MAX_CHARS` | `128000` | Cap on a `read` tool result (chars); over-cap reads truncate + a notice that teaches the grep → `read(offset, limit)` workflow (phase 95, rewritten phase 131) |

### Import & sources

| Env | Default | Meaning |
|-----|---------|---------|
| `BOR_IMPORT_EXTENSIONS` | A9 family | Importable text formats — the A9 default is `md,markdown,txt,yaml,yml,json,py,container,network,volume,image,pod,kube,swap,os,endpoint,j2`; any well-formed extension is allowed. Add the `pdf` token to make PDFs first-class, retrievable documents (phase 126, opt-in). Images are **not** added here — see `BOR_IMAGES` |
| `BOR_PDF_PICTORIAL` | `1` | Master switch for the pictorial PDF pass (phase 140, on by default): a usable-text PDF's sparse pages (stripped text under `BOR_PDF_PICTORIAL_MIN_CHARS`) are rendered + vision-described into the indexed content (`[Page N illustration: …]`); `0` ⇒ content byte-identical to pre-phase-140 (no render, no vision call) |
| `BOR_PDF_PICTORIAL_MIN_CHARS` | `100` | Sparseness bar for the pictorial pass (phase 140): a page whose STRIPPED text is STRICTLY shorter than this many chars is a candidate (a page with exactly this many is not); `0` = every page (A21); a negative value fails startup loudly |
| `BOR_PDF_IMAGE_SKIP_BLANK` | `1` | Skip blank/decorative images embedded in PDFs (phase 151, on by default): near-white, nearly-uniform images (the table-shading "blank white" underlays) are dropped from the embedded-image pass — no row, no copy, no vision call; `0` ⇒ every extracted image is indexed again (pre-phase behavior; dedupe + soft-mask compositing unaffected) |
| `BOR_PDF_PAGE_DIR` | `~/bor-sources/pages` | Persistent home for the rendered PDF page PNGs (phase 148): a page-qualifying PDF (the phase-133 screenshot class or the phase-140 pictorial class) gets every page rendered at index time as `<doc-id>-p{N}.png` here, so the document viewer shows the actual pages (the descriptions stay embedded for Q&A, just not displayed); text-heavy PDFs keep the text view. Clearing the directory is self-healing — the next sync re-gates any strip that still claims pages and re-renders what is missing, with zero vision calls (phase 163) |
| `BOR_GIT_SOURCES` | — (empty) | CSV of git repo URLs — **fallback while the admin Git sources page's list is empty** |
| `BOR_SOURCES_DIR` | `~/bor-sources` | Where git repos are cloned/pulled |
| `BOR_UPLOAD_DIR` | `~/bor-sources/uploads` | Where uploaded source archives are unpacked |
| `BOR_UPLOAD_MAX_MB` | `512` | Cap (MiB) for uploaded source archives (zip-bomb guard) |
| `BOR_SUGGESTIONS` | built-in list | JSON seed for onboarding chips (shown before the first saved question) |

### Image support (phases 122/123/124)

| Env | Default | Meaning |
|-----|---------|---------|
| `BOR_IMAGES` | `0` | Master switch for image support — **off by default; enable only when `BOR_LLM_CHAT_MODEL` supports vision**. Gates both image documents and question images |
| `BOR_IMAGE_EXTENSIONS` | `png,jpg,jpeg,webp,gif,bmp` | Extensions indexed as image documents (case-insensitive; separate from `BOR_IMPORT_EXTENSIONS`) |
| `BOR_IMAGE_DIR` | `~/bor-sources/images` | Persistent home for the served image bytes (uploads are replaced, checkouts re-cloned) |
| `BOR_CHAT_IMAGE_DIR` | `~/bor-sources/chat-images` | Where question images are stored (saved/shared chats carry the path, never base64) |
| `BOR_CHAT_IMAGE_MAX_MB` | `10` | Question-image upload cap (MiB; must be > 0) |
| `BOR_ANSWER_IMAGES` | `0` | Let the chat model show a standalone image document as part of its answer (phase 124): offers the agent the `show_image` tool and renders the shown image as an answer figure — **independent of `BOR_IMAGES`** (that toggle indexes image docs; this one shows them) |

### Docs push

| Env | Default | Meaning |
|-----|---------|---------|
| `BOR_DOCS_REPO` | — (empty) | The git repo chat answers can be committed to (URL or local path); while empty "Save as doc" is hidden and the push endpoint 409s |
| `BOR_DOCS_BRANCH` | `bor-docs` | Branch pushes land on (`git push --ff-only` — you open the PR yourself) |
| `BOR_DOCS_BASE_BRANCH` | `main` | Branch each push is cut fresh from |
| `BOR_DOCS_WORK_DIR` | `~/bor-docs` | Where `BOR_DOCS_REPO` is checked out on the server |

### Admin & debugging

| Env | Default | Meaning |
|-----|---------|---------|
| `BOR_ADMIN_PASSWORD` | *(required)* | The single admin's password — app refuses to start when empty |
| `BOR_SESSION_SECRET` | *(required)* | Signing key for the `bor_session` cookie |
| `BOR_SESSION_MAX_AGE` | `43200` | Session-cookie lifetime in seconds (12 h, sliding) |
| `DEBUGPY` | `0` | `1` ⇒ attach-on-demand debugpy on `DEBUGPY_PORT` |
| `DEBUGPY_PORT` | `5678` | debugpy listen port |

### Customizing the look

Every identity string is an env var: `BOR_APP_NAME` (display name),
`BOR_INPUT_PLACEHOLDER` (chat composer placeholder), and
`BOR_FOOTER_TEXT` (footer line) — all served by `GET /api/config` and
applied by `assets/brand.js` at boot.

The colors are set from the admin **Theme tab** (`/theme.html`,
admin-only — phase 91): the 17 palette colors (the 9 identity colors plus
the 8 semantic state colors — success/error/accent families) and the
three strings above are edited with text fields and color pickers,
persisted in the `ui_settings` table, and injected into every served page
as an inline `<style>` BEFORE first paint (no red flash, no pop-in). The old
CSS-file theming is retired — a leftover `BOR_THEME` line in a
deployment's `.env` is simply ignored. Leave everything unset and the
app renders the built-in dark-tech palette byte-identically.

---

## Checking retrieval quality

Ask the *real* pipeline (live aipi embeddings + the current KB) whether a
question lands on the right document, with gate verdict and per-document
cosine / FTS / fused scores:

```bash
uv run python -m scripts.eval_retrieval "How did I install gitlab?"
uv run python -m scripts.eval_retrieval --from-file questions.txt --top 8
```

Requires an aipi key — `BOR_LLM_API_KEY` in `.env` (the stock
configuration) or `$AIPI_KEY` in the environment — and an imported
knowledge base.

---

## Debugging

`debugpy` is **off by default** and *never imported* unless you opt in:

```bash
DEBUGPY=1 uv run uvicorn app.main:app
# → log line: debugpy: remote debugging ENABLED, listening on 0.0.0.0:5678
```

Attach from VS Code (`.vscode/launch.json`):
```json
{
  "name": "Attach to Brain of Reese",
  "type": "debugpy",
  "request": "attach",
  "connect": { "host": "localhost", "port": 5678 },
  "pathMappings": [
    { "localRoot": "${workspaceFolder}", "remoteRoot": "/app" }
  ]
}
```

The port is non-blocking and attach-on-demand: the app keeps running
normally until you attach. Override the port with `DEBUGPY_PORT`.

---

## QA / Testing

Three layers — the project rule is **one phase, one Playwright suite** (a
phase may implement a user story or an owner-directed change; see
`AGENTS.md`):

```bash
# Unit + integration (FastAPI TestClient)
uv run pytest

# Coverage gate (phases require >90% on app/)
uv run pytest --cov=app --cov-report=term-missing

# Lint + static types
uv run ruff check .
uv run pyright

# Playwright E2E — install the browser once:
uv run playwright install chromium

# Each phase's E2E runs IN ISOLATION:
uv run pytest tests/e2e/test_import_documents.py -v --no-cov
uv run pytest tests/e2e/test_chat_rag.py -v --no-cov
# ...one file per phase in tests/e2e/ (the stories live in
# .agents/user_stories/, see .agents/phases/todo/)
```

**Retrieval regression battery.** `tests/fixtures/retrieval_battery.txt` is
the tracked retrieval gate (phases 119 + 162): the name-your-tool questions,
the versioned-name (`qwen 3.8 27b`) regression, the out-of-KB deflection
sanity, and the `besta` / `BESTÅ` accent-fold pair. Run it against a KB with
`scripts.eval_retrieval` (the real pipeline, real embeddings):

```bash
uv run python -m scripts.eval_retrieval --from-file tests/fixtures/retrieval_battery.txt
```

The exact expected ranks are recorded in the fixture's header comments; a
question whose source is missing from the KB snapshot is recorded as skipped
loudly, never silently dropped.

**Deterministic E2E:** by default the E2E app talks to a local **mock
aipi** (`tests/e2e/mock_llm.py`) whose embeddings are real token-overlap
vectors — so the cosine relevance threshold behaves like production. To
run E2E against the **live** self-hosted models instead:

```bash
E2E_REAL_LLM=1 uv run pytest tests/e2e/test_chat_rag.py -v --no-cov
```

---

## Production Deployment

Build the multi-stage image (frontend minified by esbuild in the builder
stage, deps installed by `uv`, non-root runtime):

```bash
podman build -t brain-of-reese/app:latest .
```

Run standalone (bring your own Postgres + pgvector):
```bash
podman run -d --name brain-of-reese \
  -p 8000:8000 \
  -e BOR_DATABASE_URL=postgresql+psycopg://reese:SECRETPASSWORD@dbhost:5432/brain_of_reese \
  -e BOR_LLM_BASE_URL=https://aipi.reeseapps.com/v1 \
  -e BOR_LLM_API_KEY=$AIPI_KEY \
  -e BOR_ADMIN_PASSWORD=your-password \
  -e BOR_SESSION_SECRET=<the-hex-from-the-setup-above> \
  brain-of-reese/app:latest
```

`BOR_ADMIN_PASSWORD` and `BOR_SESSION_SECRET` are **required** — the
fail-loud boot gate in [Admin & Sign-in](#admin--sign-in) refuses to
start while either is empty.

The entrypoint runs `alembic upgrade head` automatically on start.

Or run the whole stack from compose (app + db):
```bash
podman compose --profile prod up -d --build
```
The `app` service picks up `BOR_ADMIN_PASSWORD` / `BOR_SESSION_SECRET`
from the shell environment (or an env file) — the same fail-loud gate
applies.

Production hardening: app runs as non-root (uid 10001), slim image,
healthcheck on `/api/health`, debugpy off unless `DEBUGPY=1`, all assets
served locally (no CDN), `BOR_ENVIRONMENT=production`.

---

## Troubleshooting

- **`401` from aipi** — set `BOR_LLM_API_KEY` (or `$AIPI_KEY`).
- **`litellm.UnsupportedParamsError … encoding_format`** — the aipi proxy
  (litellm `openai_like`) rejects the `encoding_format` parameter. The app
  already works around this by POSTing a minimal `{model, input}` payload.
  If you see this, you are likely calling the endpoint with a different
  client — drop the parameter (or set `litellm.drop_params = True` on the proxy).
- **Embedding dimension mismatch** — aipi changed models; run
  `uv run python -m scripts.llm_probe`, update `BOR_EMBEDDING_DIM`, then
  drop + recreate the chunks table.
- **No attach button / images ignored on import** — image support is
  off (the default). Set `BOR_IMAGES=1` in `.env` and restart — only
  when `BOR_LLM_CHAT_MODEL` supports vision. Turning the toggle off
  never destroys existing image documents (syncs prune-guard them).
- **Honest deflection** (the amber deflection bubble) — every question
  passes the honesty gate: deflection happens
  when the best cosine similarity is below `BOR_RELEVANCE_THRESHOLD`
  (default `0.62`) **and** the lexical signal is absent or not
  vector-corroborated (`fts_hits = 0`, or best cosine below
  `BOR_LEXICAL_SUPPORT_FLOOR` — default `0.35` — A8 revised 2026-09-14).
  A weak cosine with a lexical hit (name-your-tool questions) still gets
  a grounded answer, provided the best cosine clears the floor. When it
  does deflect, the LLM prompt carries weak-hit *titles only* (no document
  content), the reply admits it has no notes on that and offers 2-3
  concrete alternative questions about things it **does** have notes on,
  the bubble renders amber with *"Maybe try"* chips derived from the
  closest indexed titles, and the `query_log` row records
  `deflected=true`. This is a feature, not a bug. With a small local
  model, a rare turn may still answer from general knowledge with an
  explicit disclosure when retrieval was borderline — the gate (A8
  revised) minimizes this by keeping vector-unsupported docs out of the
  grounded prompt, and the disclosure is surfaced, never silent.
- **Answers deflect too often / too rarely** — tune
  `BOR_RELEVANCE_THRESHOLD` (lower = answers more, higher = more honest
  deflection) and `BOR_LEXICAL_SUPPORT_FLOOR` (the best-cosine floor a
  lexical hit must clear to flip HIGH — lower = lexical hits ground more
  easily): `0.0` ⇒ the gate leans entirely on FTS hits; `1.0` ⇒
  everything deflects unless a lexical hit is vector-corroborated. The
  `embed` model's cosines cluster in a ~0.6–0.85 band on the live KB, so
  the default is `0.62`. Check real scores:
  ```sql
  SELECT question, top_score, fts_hits, deflected
  FROM query_log ORDER BY created_at DESC LIMIT 20;
  ```
- **KB offline banner in the chat** — Postgres isn't running:
  `podman compose up -d db`.
- **Stuck "Thinking…"** — the LLM is slow or down; the client's 300 s
  pre-token guard (`TURN_TIMEOUT_MS`) turns a hung stream into the error
  state automatically.

---

## Planning & Architecture

The architecture, LOCKED decisions, and the phase roadmap live in
[`.agents/PLAN.md`](.agents/PLAN.md); per-story specs in
[`.agents/user_stories/`](.agents/user_stories/).