brain-of-reese/README.md

# 🧠 Brain of Reese

A chippy, honest **RAG chatbot** over the `~/Homelab` and `~/Deployments`
projects. Point it at your notes — markdown, YAML, JSON, Python, plain
text — ask it anything, and it retrieves the relevant chunks with
**hybrid search** (pgvector cosine ∪ Postgres full-text search, fused with
Reciprocal Rank Fusion), feeds the **whole relevant document** to a
**self-hosted LLM** (`turbo` via `https://aipi.reeseapps.com/v1`), and
streams a grounded answer back.

If it doesn't have notes for your question, it admits it:
*"I haven't done anything like that"* — plus suggestions for what it **does** know.

> **Updated your notes?** The knowledge base is refreshed by re-running the
> import — it's idempotent and only re-embeds what changed:
> ```bash
> uv run python -m scripts.import_docs --prune
> ```
> Details in [Updating the documents](#updating-the-documents).

- **Stack:** FastAPI · Pydantic v2 · SQLAlchemy 2 · Alembic · pgvector ·
  vanilla HTML/CSS/JS (no CDN) · Playwright E2E
- **Planning:** architecture, LOCKED decisions and the phase roadmap live
  in [`.agent/PLAN.md`](.agent/PLAN.md); per-story specs in
  [`.agent/user_stories/`](.agent/user_stories/).

---

## Development Setup

### Prerequisites
- [uv](https://docs.astral.sh/uv/)
- [Podman](https://podman.io/) (with the `podman compose` provider)
- Node.js is **not** needed locally (asset minification happens in the
  container build only)

### 1. Install dependencies
```bash
uv sync
```

### 2. Configure
```bash
cp .env.example .env
# edit .env — the defaults already match the local compose setup.
# BOR_LLM_API_KEY: your aipi key (falls back to $AIPI_KEY if unset)
```

### 3. Start the database (Postgres 17 + pgvector)
```bash
podman compose up -d db
podman compose ps          # wait until "healthy"
```

### 4. Apply migrations
```bash
uv run alembic upgrade head
```

### 5. Import your knowledge base
```bash
uv run python -m scripts.llm_probe      # sanity: models + 768-dim check
uv run python -m scripts.import_docs    # import the configured sources (below)
```

Two source modes:

- **Git sources (recommended)** — set `BOR_GIT_SOURCES` in `.env` to a
  comma-separated list of git repo URLs. `import_docs` clones each repo
  (first run) or pulls it (subsequent runs) into
  `BOR_SOURCES_DIR/<repo-name>/` (default `~/bor-sources`) and indexes the
  checkouts — see [Git-based sources](#git-based-sources).
- **Manual directories** — `--source <path>` (repeatable) imports local
  directories directly and *always wins* over `BOR_GIT_SOURCES`.
- If neither is set, the import falls back to the **previous** default,
  `~/Homelab` + `~/Deployments` — kept only for backwards compatibility,
  now replaced by `BOR_GIT_SOURCES`.

### 6. Run the app
```bash
uv run uvicorn app.main:app --reload
# → http://localhost:8000  (chat)   http://localhost:8000/sources.html (KB)
```

> 📝 **After this, day-to-day is just: edit markdown → re-run the import.**
> See [Updating the documents](#updating-the-documents) below.

## Using the UI

- **Chat** (`/`) — ask questions; answers stream in with **source chips**
  that cite the exact documents used. Clicking a chip opens that document
  in an **almost-fullscreen modal** on the same page (no new tab).
- **Document viewer** — the modal above *is* the viewer; the full text of
  any indexed document is served from the database (no filesystem access):
  markdown is rendered, every other format (`yaml`, `json`, `py`, `txt`, …)
  is shown as escaped monospace text. `/document.html?source=…&path=…`
  stays as the **full-page / direct-link** form (the modal's “Full page”
  button and the URL to share — it works without JS). Unknown documents
  get a designed not-found state with a link back to the index.
- **Sources** (`/sources.html`) — the indexed document list; the *Path*
  column opens each document in the same **almost-fullscreen modal** (no
  new tab). **Admin-only** — anonymous visitors see a sign-in gate instead
  (the catalog is what the login locks; the document viewer itself stays
  open to everyone).

## Thinking

The self-hosted `turbo` model reasons before it answers. That reasoning is
streamed with the turn as `thinking` SSE events and shown in a
**collapsible "Thinking" block** above the answer bubble: it opens and
fills in live while the model thinks, tucks itself away the moment the
first answer token lands, and stays click-toggleable afterwards. Thinking
persists with the message, so a reloaded conversation restores the block
(collapsed) alongside the answer. How much the model thinks — or whether
it thinks at all — is the model's call: turns without reasoning render
exactly as before.

To hide it, set `BOR_STREAM_THINKING=0` — the `thinking` events stop
(the per-turn log line still counts `thinking_chars`).

## Admin & sign-in

Brain of Reese has exactly **one account: the admin (you)**. Signing in
unlocks the **full Sources catalog** and the **answer-tuning** controls;
everyone else stays anonymous and keeps **chat** and the **document
viewer** (any document an answer cites can be opened by its direct URL —
the catalog is gated, not the viewer).

### Setup (one-time)

```bash
python -c 'import secrets;print(secrets.token_hex(32))'   # → paste into .env
```

```env
BOR_ADMIN_PASSWORD=your-password        # plaintext — homelab scope, by design
BOR_SESSION_SECRET=<the hex from above> # signs the session cookie
```

**Fail-loud:** while either variable is empty the app refuses to start,
naming the missing one(s):

```
RuntimeError: Brain of Reese cannot start: admin auth is not configured.
Set the missing variable(s): BOR_ADMIN_PASSWORD, BOR_SESSION_SECRET …
```

### How it works

- `POST /api/login {"password": …}` → `204` + signed `bor_session` cookie
  (Starlette `SessionMiddleware` — an itsdangerous-signed cookie, no
  server-side store, no new service, no DB table); any mismatch → `401`
  `{"detail": "invalid password"}` (constant-time compare, one generic
  message — no user enumeration, there is only one user).
- `POST /api/logout` → `204` (session cleared and cookie expired;
  idempotent for anonymous callers).
- `GET /api/whoami` → `{"authenticated": bool, "role": "admin"|"anonymous"}`
  — the single source of truth for every UI gating decision.
- Cookie flags: `same_site="lax"`, `https_only` off — **no HTTPS
  enforcement on purpose** (homelab HTTP; the cookie is single-admin
  convenience, not a cloud boundary). Max age `BOR_SESSION_MAX_AGE`
  (default `43200` = 12 h, refreshed while active).
- Sign in from the chat header (**Sign in**) or `/login.html` directly;
  the header then offers **Sign out** (logout + reload).

### Who can do what

| Capability | Anonymous | Admin (signed in) |
|---|---|---|
| Chat (`/`) + suggestion chips | yes | yes |
| Document viewer (`/document.html?source=…&path=…`) | yes — any indexed doc by direct URL | yes |
| Sources catalog (`/sources.html`, `GET /api/docs`) | sign-in gate | full catalog |
| Tuning (Tune button, Tuning panel, `/api/steering`) | UI hidden | full |

The public API endpoints stay stateless — the signed cookie is the only
session state in the system.

## Tuning your answers

*Admin-only* — sign in first (see **Admin & sign-in** above); anonymous
visitors never see the Tune button or the Tuning panel.

If an answer isn't quite right — too chatty, wrong assumption, missing
context — **tune** Brain right there:

1. Press **“Tune”** in the meta row under any completed answer (deflected
   ones included).
2. Type a short instruction (1–2000 chars), e.g. *“be more concise”* or
   *“assume I'm on NixOS”*, and **Save**.

The note is stored in Postgres (`steering_notes`) and read into the
**system prompt of every subsequent chat turn** as a `<tuning>` section
(numbered, oldest first, capped at `BOR_STEERING_MAX_CHARS` chars —
default 8000, overflow marked `[…truncated…]`). With no stored notes the
prompt is byte-identical to the un-tuned one, so tuning is opt-in per
note.

List or remove notes at any time from the **“Tuning”** button in the chat
header (count badge, newest-first, per-note delete). The API is stateless
JSON if you prefer curl:

```bash
curl -s localhost:8000/api/steering                                  # list (newest first)
curl -s -X POST localhost:8000/api/steering \
     -H 'Content-Type: application/json' -d '{"note": "be more concise"}'
curl -s -X DELETE localhost:8000/api/steering/<note-id>              # remove
```

## Updating the documents

**This is the workflow you'll use most.** The knowledge base is refreshed by
**re-running the import**. It is idempotent and delta-based (sha256 per
file), so a refresh after a normal editing session takes seconds:

```bash
# After editing/adding/removing notes in your projects:
uv run python -m scripts.import_docs                 # re-index what changed
uv run python -m scripts.import_docs --prune         # also drop deleted/out-of-scope files

# Point it at extra directories (repeatable):
uv run python -m scripts.import_docs --source ~/SomeOtherDocs
```

With **git-based sources** (below) each run first pulls the latest commits
of your repos, so this same command is the whole update loop: commit in the
repo → re-run the import.

Then check the **Sources** page (`http://localhost:8000/sources.html`):
the *documents* / *chunks* counters and *last indexed* timestamp should
reflect the new files, and each document row shows when it was last
embedded.

- The import prints one line per file (`import: added|updated|unchanged|
  pruned …`) and ends with a greppable summary (`import: summary files=…
  added=… updated=… unchanged=… pruned=… chunks=… embed_batches=…
  formats=md:203,yaml:267,…`), so it is safe to run from a cron job or
  after every commit.
- Indexed formats (A9): **`md, markdown, txt, yaml, yml, json, py`**
  (case-insensitive; narrow with `BOR_IMPORT_EXTENSIONS`). Any path with a
  **dot-prefixed component** — hidden files or vendored caches like
  `.esphome/.espressif/**` — is skipped, along with `.venv`,
  `node_modules`, `.git`, `__pycache__`, `.pytest_cache`, `dist`, `build`.
  `--prune` also drops documents whose files no longer match the filter —
  that's how previously imported junk leaves the index.
- Non-markdown files get format-aware chunking (YAML top-level keys /
  `---` docs, JSON top-level keys, Python top-level defs/classes via
  stdlib `ast`) and their title comes from the file stem.
- Unchanged files are **not re-embedded** — only new/changed ones, so
  refreshes are cheap.
- After a run that **changed** the knowledge base (at least one document
  added or updated), the import also regenerates the stored **KB
  overview** — a plain-text outline of the KB's basic categories that
  every chat turn injects into the system prompt as `<knowledge_base>`
  (phase 31). The regeneration is best-effort and change-gated: unchanged
  re-imports and `--limit` debug runs skip it (no `lite` call), and a
  `lite`-model failure leaves the previous outline intact without failing
  the import. The summary line ends `overview=updated|skipped|failed`.
- To sanity-check the LLM backend (models + embedding dimension) after any
  aipi change: `uv run python -m scripts.llm_probe`.

### Git-based sources

Rather than pointing the import at local folders, point it at **git
repositories** — the notes live in the repos and `import_docs` keeps local
checkouts of them up to date for you:

```env
# .env
BOR_GIT_SOURCES=https://git.reeseapps.com/reese/homelab.git,git@github.com:reese/deployments.git
BOR_SOURCES_DIR=~/bor-sources   # default; each repo lands in <dir>/<repo-name>/
```

- `BOR_GIT_SOURCES` is a **comma-separated list** of URLs. Auth is whatever
  the machine supplies — `https://…` via the OS credential helper, or
  `git@host:repo.git` via your SSH key; no credentials are stored in the
  app or `.env`.
- Every run **clones** each repo (first time, shallow `--depth 1`) or
  **pulls** it (`git pull --ff-only` — fast-forward only, so a diverged or
  broken checkout fails loudly instead of merging) into
  `BOR_SOURCES_DIR/<repo-name>/`, then indexes the checkouts exactly like
  any local directory (A9 format filter, hidden-dir skip, sha256 delta).
  `documents.source` is the repo directory name (e.g. `homelab`).
- **`--source <path>` overrides**: when the flag is given,
  `BOR_GIT_SOURCES` is ignored and the manual directory(ies) are imported.
- **A failed sync aborts the run**: if any repo cannot be cloned/pulled,
  `import_docs` exits non-zero naming the failing repo and imports
  **nothing** (no partial junk). Fix the URL/connectivity and re-run — the
  other checkouts stay on disk and are pulled as usual.

## Checking retrieval quality

Ask the *real* pipeline (live aipi embeddings + the current KB) whether a
question lands on the right document, with the gate verdict and per-document
cosine / FTS / fused scores:

```bash
uv run python -m scripts.eval_retrieval "How did I install gitlab?"
uv run python -m scripts.eval_retrieval --from-file questions.txt --top 8
```

Requires `AIPI_KEY` in the environment (same convention as
`scripts/llm_probe.py`) and an imported knowledge base.

## How retrieval works (hybrid)

Every question is embedded and also lexically tokenized (OR-joined, English
stemming) and searched **twice** against Postgres:

1. **Vector** — pgvector cosine top-N (default `BOR_HYBRID_VECTOR_CANDIDATES=100`)
2. **Lexical** — a stored `tsvector` (GIN-indexed) matched with `to_tsquery`,
   top-N by `ts_rank` (default `BOR_HYBRID_LEXICAL_CANDIDATES=30`)

The two ranked lists are fused with **Reciprocal Rank Fusion**
(`score = Σ 1/(k + rank)`, `BOR_RRF_K=60`) — a chunk in both lists scores
nearly double, which is what lets a name-your-tool question ("gitlab") find
its own document even when the question embeds close to generic templates.

The **honesty gate** (A8) then answers (HIGH) when the best cosine is ≥
`BOR_RELEVANCE_THRESHOLD` (default `0.62`) **or** at least one chunk matched
lexically (`fts_hits > 0`) — it deflects (LOW) only when *both* signals are
absent. The top `BOR_TOP_N_DOCS` full documents are still what the LLM sees.

`query_log` records every turn (`top_score` = best cosine, `fts_hits`,
`chunk_hits`, `deflected`, `sources`, `latency_ms`) — the raw material for
tuning: `psql … -c 'SELECT question, top_score, fts_hits, deflected FROM query_log ORDER BY created_at DESC LIMIT 20'`.

## Document summaries (non-markdown)

Raw yaml/json/py/txt embeds badly — flags and keys are not language, so
retrieval can miss exactly the documents that are all configuration. At
import time, every **non-markdown** A9 document is summarized by the aipi
`lite` model (one-shot `LLMClient.chat`, `BOR_LLM_SUMMARY_MODEL`, default
`lite`):

* the summary is stored on `documents.summary` **and** indexed as one
  extra embedded chunk (`chunks.is_summary`, position −1), so hybrid
  search has a natural-language target to hit instead of the raw text;
* the last line is a **code-deterministic** pointer — `Source:
  <source>/<path>` — appended by the app, never model-generated;
* the model only sees the first `BOR_SUMMARY_MAX_CHARS` (default 12000)
  characters of the document; overflow is cut and marked with the shared
  `[…truncated…]` marker.

A summary hit resolves through the normal chunk→document mapping: the
LLM receives the **full source document** (never the summary alone,
never truncated). Markdown files are natural language already and get no
summary. Summary generation is best-effort: if `lite` fails for a file,
the document is still indexed (without a summary), the failure is logged
and counted in the import summary line (`summaries=N summary_errors=N`).
On a chat turn, the per-turn log line records `summary_hits=N` — how many
summary chunks of the selected context the question landed on.

## Debugging

`debugpy` is **off by default** and *never imported* unless you opt in —
zero overhead in normal runs.

```bash
DEBUGPY=1 uv run uvicorn app.main:app
# → log line: debugpy: remote debugging ENABLED, listening on 0.0.0.0:5678
```

Then attach from VS Code (`.vscode/launch.json`):
```json
{
  "name": "Attach to Brain of Reese",
  "type": "debugpy",
  "request": "attach",
  "connect": { "host": "localhost", "port": 5678 },
  "pathMappings": [
    { "localRoot": "${workspaceFolder}", "remoteRoot": "/app" }
  ]
}
```
The port is non-blocking and attach-on-demand: the app keeps running
normally until you attach. Override the port with `DEBUGPY_PORT`.

## QA / Testing Environment

Three layers — the project rule is **one story, one phase, one Playwright
suite** (see `AGENTS.md`):

```bash
# Unit + integration (FastAPI TestClient)
uv run pytest

# Same, with the coverage gate (phases require >90% on app/)
uv run pytest --cov=app --cov-report=term-missing

# Lint + static types
uv run ruff check .
uv run pyright

# Playwright E2E — install the browser once:
uv run playwright install chromium

# Each story's E2E runs IN ISOLATION (DB must be up):
podman compose up -d db
uv run pytest tests/e2e/test_import_documents.py -v --no-cov
uv run pytest tests/e2e/test_chat_rag.py -v --no-cov
# ...one file per story in .agent/user_stories/ (see .agent/phases/todo/)
```

**Deterministic E2E:** by default the E2E app talks to a local **mock
aipi** (`tests/e2e/mock_llm.py`) whose embeddings are real
token-overlap vectors — so the cosine relevance threshold behaves like
production (on-topic questions answer, off-topic ones deflect).
To run E2E against the **live** self-hosted models instead:

```bash
E2E_REAL_LLM=1 uv run pytest tests/e2e/test_chat_rag.py -v --no-cov
```
(requires a real import of your docs first).

## Production Deployment

Build the multi-stage image (frontend minified by esbuild in the builder
stage, deps installed by `uv`, non-root runtime):

```bash
podman build -t brain-of-reese/app:latest .
```

Run standalone (bring your own Postgres + pgvector):
```bash
podman run -d --name brain-of-reese \
  -p 8000:8000 \
  -e BOR_DATABASE_URL=postgresql+psycopg://reese:SECRETPASSWORD@dbhost:5432/brain_of_reese \
  -e BOR_LLM_BASE_URL=https://aipi.reeseapps.com/v1 \
  -e BOR_LLM_API_KEY=$AIPI_KEY \
  brain-of-reese/app:latest
```
The entrypoint runs `alembic upgrade head` automatically on start.

Or run the whole stack from compose (app + db):
```bash
podman compose --profile prod up -d --build
```

Production hardening notes: app runs as non-root (uid 10001), slim image,
healthcheck on `/api/health`, debugpy off unless `DEBUGPY=1`, all assets
served locally (no CDN), `BOR_ENVIRONMENT=production`.

## Configuration reference

| Env | Default | Meaning |
|-----|---------|---------|
| `BOR_DATABASE_URL` | local compose URL | SQLAlchemy URL (psycopg) |
| `BOR_LLM_BASE_URL` | `https://aipi.reeseapps.com/v1` | OpenAI-compatible endpoint |
| `BOR_LLM_API_KEY` | — (falls back to `$AIPI_KEY`) | aipi API key |
| `BOR_LLM_CHAT_MODEL` | `turbo` | chat model |
| `BOR_LLM_EMBED_MODEL` | `embed` | embedding model |
| `BOR_LLM_SUMMARY_MODEL` | `lite` | one-shot (non-streaming) completions: document summaries at import (phase 30) and the KB overview (phase 31) |
| `BOR_EMBEDDING_DIM` | `768` | vector dimension (fixed at table creation) |
| `BOR_TOP_N_DOCS` | `2` | full documents fed to the LLM |
| `BOR_RELEVANCE_THRESHOLD` | `0.62` | answer when best cosine ≥ this **or** an FTS hit; below + no FTS ⇒ honest deflection |
| `BOR_HYBRID_VECTOR_CANDIDATES` | `100` | cosine list width for the RRF fusion |
| `BOR_HYBRID_LEXICAL_CANDIDATES` | `30` | FTS list width for the RRF fusion |
| `BOR_RRF_K` | `60` | RRF damping constant (`1/(k + rank)`) |
| `BOR_IMPORT_EXTENSIONS` | `md,markdown,txt,yaml,yml,json,py` | csv of importable formats (may only narrow the A9 set) |
| `BOR_GIT_SOURCES` | — (empty) | csv of git repo URLs; `import_docs` clones/pulls them into `BOR_SOURCES_DIR` and indexes the checkouts (see *Git-based sources*) |
| `BOR_SOURCES_DIR` | `~/bor-sources` | where the `BOR_GIT_SOURCES` repos are cloned/pulled (one subdirectory per repo) |
| `BOR_STEERING_MAX_CHARS` | `8000` | char budget for the `<tuning>` (steering notes) prompt section |
| `BOR_SUMMARY_MAX_CHARS` | `12000` | cap on document content sent to the `lite` summary model at import (see *Document summaries*) |
| `BOR_KB_OVERVIEW_MAX_CHARS` | `4000` | char budget for the `<knowledge_base>` (KB overview) prompt section |
| `BOR_OVERVIEW_INPUT_MAX_CHARS` | `40000` | cap on the document list sent to the `lite` model when generating the KB overview |
| `BOR_SUGGESTIONS` | built-in list | JSON list of onboarding chips |
| `BOR_ADMIN_PASSWORD` | *(required)* | the single admin's password (plaintext, `.env`); app refuses to start when empty |
| `BOR_SESSION_SECRET` | *(required)* | signing key for the `bor_session` cookie; `python -c 'import secrets;print(secrets.token_hex(32))'` |
| `BOR_SESSION_MAX_AGE` | `43200` | session-cookie lifetime in seconds (12 h, sliding) |
| `DEBUGPY` | `0` | `1` ⇒ attach-on-demand debugpy on `DEBUGPY_PORT` (default 5678) |
| `BOR_LOG_LEVEL` | `INFO` | app log level |

## Troubleshooting

- **`401` from aipi** — set `BOR_LLM_API_KEY` (or `$AIPI_KEY`).
- **`litellm.UnsupportedParamsError … encoding_format` from aipi** — the
  aipi proxy (litellm `openai_like`) rejects the `encoding_format` parameter
  that the `openai` SDK injects into every embeddings request. The app
  already works around this by POSTing a minimal `{model, input}` payload
  through the openai client's own httpx transport (`app/rag/llm.py` →
  `LLMClient._embed_batch`). If you see this, you are likely calling the
  endpoint with a different client — drop the parameter (or set
  `litellm.drop_params = True` on the proxy).
- **Embedding dimension mismatch** — aipi changed models; run
  `uv run python -m scripts.llm_probe`, update `BOR_EMBEDDING_DIM`, then
  drop + recreate the chunks table (new migration or manual `TRUNCATE
  chunks, documents`).
- **Honest deflection (the amber “I haven't done anything like that”
  bubble)** — every question passes the honesty gate: deflection happens
  only when the best cosine similarity is below
  `BOR_RELEVANCE_THRESHOLD` (default `0.62`) **and** no chunk matched the
  question lexically (`fts_hits = 0`). A weak cosine with a lexical hit
  (name-your-tool questions) still gets a grounded answer. When it does
  deflect, the LLM prompt carries weak-hit *titles only* (no document
  content), the reply opens with “I haven't done anything like that”, the
  bubble renders amber with “Maybe try” chips derived from the closest
  indexed titles, the SSE `done` event carries `deflected: true` +
  `suggestions[]`, and the `query_log` row records `deflected=true` + the
  weak `top_score` + `fts_hits`. This is a feature, not a bug — the KB
  simply has no notes that close; the chips always point at topics Brain
  really covers.
- **Answers deflect too often / too rarely** — tune
  `BOR_RELEVANCE_THRESHOLD` (lower = answers more, higher = more honest
  deflection): `0.0` ⇒ the gate leans entirely on FTS hits; `1.0` ⇒
  everything deflects unless a chunk matches lexically. The `embed` model's
  cosines cluster in a ~0.6–0.85 band on the live KB, so the default is
  `0.62`; after changing it, check the real scores:
  `psql … -c 'SELECT question, top_score, fts_hits, deflected FROM query_log ORDER BY created_at DESC LIMIT 20'`
- **KB offline banner in the chat** — Postgres isn't running:
  `podman compose up -d db`.
- **Stuck "Thinking…"** — the LLM is slow or down; a 120s client timeout
  turns it into an error banner automatically.