# Brain of Reese — Master Plan (minimal)

> **Status:** Minimal re-land (2026-09-10). Phases 01–92 shipped;
> `todo/` holds 93–95 (see §12). Section numbers and anchor IDs match the
> previous full plan (in git history, commit `dac4a3e`) so the codebase's
> `PLAN §…` / anchor comments stay valid — consult that revision or the
> phase records in `.agents/phases/complete/` for full detail and the
> complete owner-permission revision log.
>
> **Rule:** Every agent reads this file first. `LOCKED` anchors in §2 are
> settled — a change requires explicit owner permission, recorded as a
> dated revision note under the table. Never a silent deviation
> (AGENTS.md rule 3).
>
> **This is deliberately minimal:** it captures the tech stack, the
> architecture an agent needs to work, and the locked decisions. It does
> NOT recap the shipped work — the git log, the README, and the completed
> phase directories are the history.

---

## 1. Mission

A **knowledge-base chatbot** ("Brain of Reese") over the owner's homelab
documentation — git repos, local directories, and uploaded archives
registered on the admin Sources page — embedded into Postgres + pgvector.
Product feel: chippy, upbeat, and **radically honest** — if retrieval
surfaces nothing relevant it says *"I haven't done anything like that"*
and offers alternatives instead of hallucinating. Self-hosted LLMs only.

**Deliberately out of scope:** binary/non-text ingestion; real-time file
watching (the import script / Sync button is the refresh loop); PR
tooling for docs push (the owner opens the PR); multi-admin/per-user
accounts (one admin + hand-out tokens is the model).

---

## 2. Tech Stack & Architectural Anchors (LOCKED)

| # | Component | Decision |
|---|-----------|----------|
| A1 | Runtime | Python 3.12+, `uv` for all package management (API + scripts + tests, one venv) |
| A2 | Web framework | FastAPI + Pydantic v2 + Uvicorn (async, SSE-friendly) |
| A3 | Database | **PostgreSQL 17 + pgvector** (local build of official `postgres:17`, `db/Containerfile`), cosine (`<=>`) search; one system for relational + vectors |
| A4 | Orchestration | `compose.yaml`, `podman compose up -d` (dev: `db` only; `--profile prod` adds the app container) |
| A5 | LLM backend | OpenAI-compatible self-hosted endpoint `https://aipi.reeseapps.com/v1` via the `openai` async client: **`turbo`** (chat, streams `reasoning_content` thinking), **`embed`** (embeddings), **`lite`** (one-shot: document summaries, KB overview; phase 94 adds sync-time folder summaries) |
| A6 | Embedding dim | **768** (verified against the live endpoint); `chunks.embedding` is fixed at table creation — a dim mismatch must **fail loudly**, never silently re-embed |
| A7 | Retrieval→context | **Hybrid:** cosine top-100 ∪ Postgres FTS top-30 (OR tsquery, `ts_rank`), fused with **RRF (k=60)** → parent docs ranked by best fused chunk → **full text of top-N=2 documents, never truncated** on the retrieval path (owner 2026-08-24: "this should never happen"; the agent `read`-tool cap is a separate owner-permitted path — phase 95) |
| A8 | Honesty gate | Deflect (LOW mode) **only when** best cosine < `BOR_RELEVANCE_THRESHOLD` (0.62) **and** zero FTS hits; LOW prompt carries weak-hit *titles only* + the `DEFLECT_MODE` marker (the E2E mock keys on its presence) + the plain-text no-tools line |
| A9 | Content scope | Default `md, markdown, txt, yaml, yml, json, py` + Podman quadlet family + `j2`; `BOR_IMPORT_EXTENSIONS` may name **any** well-formed extension or narrow the set; hidden (dot) paths + exclusion list + per-source `ignore_paths` (raw prefixes, no globs) always apply |
| A10 | State & auth | `POST /api/chat` is **stateless** (client-provided `history` only, budget-trimmed — nothing stored per conversation). Auth = single admin, signed `bor_session` cookie (Starlette `SessionMiddleware` + itsdangerous; no server-side session store); admin-issued SHA-256-hashed access tokens are the only other identity; **only** shared chats stay anonymous. Fail-loud at boot while `BOR_ADMIN_PASSWORD`/`BOR_SESSION_SECRET` are empty |
| A11 | Frontend | Vanilla HTML/CSS/JS in git, **no CDN** — everything served by FastAPI `StaticFiles`; system font stack; the navbar views are views of ONE shell document (`frontend/index.html` + `router.js` deep-links, phase 76) |
| A12 | Aux services | **None** (no Valkey/queue/SeaweedFS): sync runs in-process, the login rate limit is in-memory per-process, sessions are the signed cookie. Restart clearing in-memory state is accepted |
| A13 | Migrations | Alembic + SQLAlchemy 2.0 (sync) + psycopg 3; **every migration ships a tested downgrade** (A13 — reversible) |
| A14 | Debugging | `debugpy` imported **only when `DEBUGPY=1`** (`app/core/debugging.py`); never imported otherwise (unit-tested) |
| A15 | Chat transport | **SSE** from `POST /api/chat`; event types `thinking`, `delta`, `tool`, `retry`, `done`, `error` (+ `tool_result`, owner-permitted extension — phase 95, not yet landed); no proxy buffering, no client caching on the stream |
| A16 | Testing | Per phase: unit + integration (pytest) with **>90% coverage on `app/`** + one dedicated **Playwright E2E file**, run **in isolation** (`--no-cov`); E2E uses a deterministic mock LLM by default (`E2E_REAL_LLM=1` opts into live aipi) |
| A17 | Git | Conventional Commits, **always `--no-gpg-sign`**, one atomic commit per completed phase |
| A18 | Docs push | Save-a-answer-as-doc: server-side draft (`doc_drafts`, long body never in a URL — unguessable `uuid4` token is the URL credential) pushed to a **generic git remote** (`BOR_DOCS_REPO`, URL or local path; no `gh`) on a dedicated branch, `--ff-only`, re-cut per push; **no PR tooling**; inert (hidden + 409) while the repo is unset |
| A19 | Deploys & caching | HTML pages `Cache-Control: no-cache` (no etag/last-modified); `/assets/*` versioned (`?v=<token>`) + `immutable, max-age=1y`; **the token is the deploy** (git `HEAD` short SHA; content-hash fallback) — a deploy is a commit the browser sees without a hard refresh |
| A20 | Security headers | Every response: `Content-Security-Policy: default-src 'self'` (+ the theme's inline-`<style>` hash on themed pages), `frame-ancestors 'none'`, `X-Content-Type-Options: nosniff`; login rate-limited 10 attempts / 15 min (in-memory, A12) |

**UI/theming anchors (B):**

| # | Decision |
|---|----------|
| B1 | Theme `ui_settings` columns: **NULL/empty = "use the default"** — env value for the 3 strings, built-in palette for colors (colors have no env fallback) |
| B3 | **Revised (owner 2026-09-10, phase 93):** the semantic state families (`--ok-*`, `--err-*`, `--accent-*`) **join** the storable palette — 17 variables total, all `NULL = built-in` (the built-in theme stays byte-identical). The 2026-09-09 lock that they are "not identity" is lifted |
| B4 | **Byte-identical contract:** with no `ui_settings` row (or a fully default theme) the served HTML is byte-identical to the built-in default — no `#bor-theme` tag, caching rewrites are rewrite-only |
| B5 | Admin-only nav views (Sources, Git sources, Tuning, History, Tokens, Theme) are hidden from non-admins; a monochrome theme must keep every state **text** label ("text + color, never color alone") |

**Key recent revisions (full log in the previous plan revision / phase
records):** A10 — single-admin cookie auth (16), saved chats + shares +
staleness (50/51/53), client history budgets (74), access tokens (79).
A15 — `thinking` (17), `tool` (37), unlimited tool rounds under
`BOR_AGENT_MAX_ROUNDS` (45), `retry` (67), harness-aligned
`ls`/`read`/`grep` surface (70). A7 — the **retrieval** path's
never-truncated contract is unchanged; the phase-95 `read`-tool cap is
the only exception, owner-permitted 2026-09-10.

---

## 3. High-Level Architecture

```
Browser (SPA shell, no CDN)
  │  HTTP pages · SSE chat · JSON API
  ▼
FastAPI app (app/main.py)
  ├── middleware: SessionMiddleware → caching (A19) → SecurityHeaders (A20)
  ├── /api/*  chat, docs, git-sources, steering, sync, chats,
  │           doc-drafts, tokens, ui-settings, auth, health, suggestions
  ├── RAG pipeline (app/rag/)  embed → retrieve → gate → agent → turbo
  └── StaticFiles (frontend/)  — shell routes registered before the mount
  │            SQL (psycopg)                     OpenAI-compat
  ▼            ▼                                 ▼
Postgres 17 + pgvector              aipi.reeseapps.com/v1
(db service, podman compose)        (self-hosted: turbo / embed / lite)

Sources registry (Postgres `git_sources`, admin-managed; DB rows win,
BOR_GIT_SOURCES is the empty-table git-only fallback):
  git repos → clone/pull via scripts/git_sync.py (the ONLY git-invocation
              site; stdlib subprocess)   local dirs → walked directly
  archives  → unpacked under BOR_UPLOAD_DIR (zip-bomb guarded)
```

| Component | Lives in |
|-----------|----------|
| App + middlewares + auth/tokens/rate-limit/theming/caching | `app/main.py`, `app/core/` |
| RAG (retriever, prompts, agent, llm, chunker, importer, summarizer, overview, scaffolding, suggestions, …) | `app/rag/` |
| API routers | `app/api/` |
| Import / sync tooling | `scripts/import_docs.py`, `app/api/sync.py`, `scripts/git_sync.py`, `scripts/eval_retrieval.py`, `scripts/llm_probe.py` |
| Frontend (shell + standalone pages) | `frontend/` (`index.html` shell; `document.html`, `shared.html`, `login.html`, `doc-edit.html`) |
| Migrations (0001–0015, all reversible) | `alembic/versions/` |

---

## 4. Chat Turn & SSE Contract (§3/§4 in older comments)

```
POST /api/chat {message, history?}   (auth: require_user)
  → embed(question)                                   [retried, phase 67 / A15 ext.]
  → cosine top-100 ∪ FTS top-30 → RRF fuse (k=60)     [A7]
  ├─ HIGH (cosine ≥ 0.62 OR fts_hits > 0):
  │    persona + <knowledge_base> + <tuning> + full top-2 <documents>
  │    + <tools> → agent loop (ls/read/grep; round-capped
  │    BOR_AGENT_MAX_ROUNDS=10, 0 = tools off) → turbo streamed
  └─ LOW (A8): DEFLECT_MODE prompt (weak titles only, no tools,
       byte-identical direct-stream path) → turbo streamed
  → query_log row + per-turn log line (§9)
```

SSE frames (`data: <json>\n\n`): `thinking` (before first delta) →
`tool` (grounded turns, one per model call) → `retry` (pre-first-frame
restarts) → `delta` (answer tokens) → `done` `{deflected, sources[],
suggestions[]}` (terminal). Failure: `error` (terminal — no `done`, no
`query_log` row; a pre-stream DB outage is a plain 503 JSON).
`BOR_STREAM_THINKING=0` suppresses `thinking` frames server-side (chars
still counted). LLM retries (phase 67, A15 extension): `BOR_LLM_RETRIES`=3 × flat
`BOR_LLM_RETRY_DELAY`=5 s, **only before a request has streamed its
first output frame**.

Retrieval details (A7/A8) and the persona/`<tools>` prompt contract: see
the previous plan revision §6 or `app/rag/retriever.py` /
`app/rag/prompts.py` / `app/rag/agent.py` — the module docstrings carry
the full contracts. **Persona text changes through the plan, not in code
(phase 03 convention); the `DEFLECT_MODE` marker and the mock-LLM markers
(`SUMMARY_MODE`, `KB_OVERVIEW_MODE`, …) may not change without updating
`tests/e2e/mock_llm.py`.**

---

## 5. Data Model & Chunking (§5 in older comments)

Tables (full column detail: `app/models.py` — it is the living doc;
migrations 0001–0015, each with a tested downgrade):

| Table | Purpose |
|-------|---------|
| `documents` | one row per imported file — full content, `(source, path)` unique, sha256 `content_hash`, `lite` `summary` (non-markdown) |
| `chunks` | retrieval units; `embedding VECTOR(768)`; `position −1` = the embedded summary chunk; stored `tsvector` (GIN) for FTS |
| `query_log` | every question: top score, FTS hits, deflection, sources, latency (threshold-tuning record) |
| `steering_notes` | owner tuning notes → `<tuning>` section of every turn (char-budgeted) |
| `kb_overview` | single row `id=1`: `lite`-generated KB outline → `<knowledge_base>` section (regenerated on KB change) |
| `git_sources` | source registry: `kind` `git`\|`local`, `url`/`path`, `ignore_paths` JSONB (phase 89) |
| `saved_chats` | owner-saved conversations; `messages` = the raw `bor.chat.v1` JSONB; `share_token` (NULL = private, `uuid4` → `/shared/<token>`); `sources_version` (stale when < current) |
| `sources_meta` | single row `id=1`: the KB generation counter — bumped once per KB-changing sync |
| `doc_drafts` | save-as-doc drafts; `token` (uuid4) is the URL credential; `draft` → `pushed` (branch + sha) |
| `api_tokens` | access tokens; only the SHA-256 of the full `bor_…` string is stored; `revoked_at` = dead |
| `ui_settings` | single row `id=1`: Theme tab persistence (3 strings + 9 identity colors; NULL = default, B1 — grows to 17 vars in phase 93) |

Single-row tables use `id = 1` (the `kb_overview` precedent).

**Chunking** (format-aware, `app/rag/chunker.py`): markdown on
`##`/`###` headings (paragraph sub-split > `BOR_CHUNK_TARGET_CHARS`=2000,
200 overlap); YAML/JSON on top-level keys; Python on top-level defs via
`ast`; txt on paragraphs; quadlet/j2 plain text. Every format honors the
**1200-char hard cap** (aipi ~1024-token request limit).

**Import workflow** (script and UI Sync share the importer): sha256
delta (unchanged files skip), two-phase upsert (one transaction per
file), `--prune` drops deleted/ignored files, non-markdown files get
`lite` summaries (best-effort/fail-soft), and a KB-changing run
regenerates the `kb_overview` + bumps `sources_meta.version` exactly
once. A failed source aborts the run — no partial junk.

---

## 6. Retrieval & Persona

(See §4's flow and the anchor rows A7/A8. The locked persona, section
order — `<relevance>` → `<knowledge_base>` → `<tuning>` → mode body —
and the `<tools>` teaching live in `app/rag/prompts.py`; the agent loop,
teaching refusals, and scaffolding filter in `app/rag/agent.py` /
`app/rag/scaffolding.py`. Empty prompt sections omit themselves — a
no-notes/no-overview prompt is byte-identical to the pre-steering text.)

Tool surface (harness-aligned, phase 70): **`ls`** (lists indexed docs
`source: X | path: Y | title: Z`; `path` = a source name — becomes a
drill-down tree in phase 94), **`read`** (combined `source/path`
including the source name; appends the full document — gets a
truncation cap in phase 95), **`grep`** (case-insensitive **fixed
substring**, ≤20 `source/path:line: text` matches, optional one-doc
scope; a locator that adds no source/context — never a regex).
Rejected calls get deterministic teaching refusals and consume a round;
`holder.tool_calls` counts executed calls only.

---

## 7. UI/UX Standards

- **Layout (§7.1):** sticky 64px header + `<main>` + footer; container
  `max-width: 72rem` centered; chat is a **centered 46rem column** (2×
  = 92rem at ≥1500px desktops — deliberate, not a bug); Sources uses a
  **full-width responsive table** — no skinny single-column lists
  (lists/tables/grids ≥80–90% of container width); document viewer is a
  near-fullscreen same-page modal; mobile ≤640px: hamburger nav,
  ≥44px touch targets, safe-area composer.
- **Accessibility (§7.2, WCAG 2.1 AA):** semantic landmarks on every
  view, labeled controls (icon buttons get `aria-label`),
  `aria-live="polite"` stream, `role="status"`/`role="alert"`,
  3px `:focus-visible`, `prefers-reduced-motion` respected, text pairs
  ≥4.5:1 (state is **text + color, never color alone**).
- **Theming (B1–B5):** built-in dark palette in
  `app/core/theming.py::BUILTIN_COLORS` (the `:root`-drift unit test
  parses built-ins from `styles.css` — no second copy); the admin Theme
  tab persists the palette in `ui_settings`, injected pre-paint as
  `<style id="bor-theme">` with a matching CSP hash (A20); **B4
  byte-identical** when nothing is set.
- **No CDN (§7.3):** zero external `<script>`/`<link>` (integration test
  on the index page); markdown rendering is a small local
  escape-first function; esbuild minify is build-time only.
- **Never-stale feedback (§7.4):** every control state — idle /
  thinking / calling tool / streaming / retrying / done (answer or
  deflected) / error / KB-offline / stopped — has a defined UI, every
  failure path re-enables its controls, a 300 s pre-token guard
  (`TURN_TIMEOUT_MS`) turns a hung stream into the error state (counts
  only visible time — phase 73), and **no auto-follow during a turn**
  (viewport moves only on user intent, phase 42). Pinned by unit tests
  on the `app.js` state machine + the story E2E suites.
- Frontend house rules: `app.js` and siblings **never build HTML
  strings** (createElement + textContent); asset paths carry
  `?v=<token>` (A19).

---

## 8. Debugging

`DEBUGPY` unset/`0` → `debugpy` never imported (unit-tested, A14).
`DEBUGPY=1` → non-blocking listener on `0.0.0.0:${DEBUGPY_PORT:-5678}`;
IDE attaches on demand. Wired at `app/main.py` module import
(`app/core/debugging.py`), so `uvicorn app.main:app`, `python -m
scripts.…`, and tests all honor it.

---

## 9. Observability

- Logs: single-line `timestamp LEVEL logger :: message` on stdout, INFO
  default (`BOR_LOG_LEVEL`), third-party loggers capped at WARNING.
- **Per-chat-turn line (required — AGENTS.md rule 10):**
  ```
  question=… embed_ms=… top_score=… fts_hits=… summary_hits=… tuning=N
  kb_chars=N history_msgs=N threshold=… deflected=… sources=…
  thinking_chars=… tool_calls=N total_ms=… retries=N scaffold_stripped=N
  ```
  (`sources=` = retrieval + agent-read docs, deduped; `tool_calls=`
  executed only; a cancelled turn writes neither line nor `query_log`
  row.)
- Importer: per-file `added|updated|unchanged|pruned` + a greppable
  cron-safe summary line (`import: summary files=… added=… … formats=…`).
- `query_log` is the durable tuning/gap-finding record.

---

## 10. Testing & Quality Gates (A16 — non-negotiable)

| Gate | Command |
|------|---------|
| Unit + integration | `uv run pytest` |
| Coverage **>90%** on `app/` | `uv run pytest --cov=app --cov-report=term-missing` |
| Story E2E, **in isolation** | `uv run pytest tests/e2e/test_<story>.py -v --no-cov` (DB up: `podman compose up -d db`) |
| Lint + types | `uv run ruff check . && uv run pyright` |

E2E determinism: `tests/e2e/mock_llm.py` is a deterministic
OpenAI-compatible mock (genuine L2-normalized token-overlap embeddings,
so the cosine gate behaves like production); `tests/e2e/slow_llm.py` is
the slow/dead variant; `E2E_REAL_LLM=1` opts into live aipi. Story E2E
fixtures truncate/re-import per module — suites must run in isolation.
Real-model tool-calling verification uses the controlled methodology in
`TOOL_CALLING_TESTING.md` + `scripts/agent_realmodel_check.py`
(fixture KB: `tests/fixtures/test_kb.dump.sql`; the 1,000-doc live-replica
snapshot restores via the `restore-test-db` skill).

---

## 11. Import & Update Workflow

```bash
uv run python -m scripts.import_docs                # sync sources + index delta
uv run python -m scripts.import_docs --prune        # also drop deleted/ignored
uv run python -m scripts.import_docs --source ~/X   # extra directories
# or one click: the admin Sources page "Sync sources" (in-process, 409 while running)
uv run python -m scripts.eval_retrieval "<question>"   # rank hybrid results (tuning)
uv run python -m scripts.llm_probe                     # models + embedding dim sanity
```

The loop for git sources is *commit → re-run*. Uploads:
`POST /api/git-sources/upload` unpacks + registers (202); the scan is
deferred to Sync (phase 90). Source removal prunes on the next sync
(phase 69).

---

## 12. Current State & Roadmap

- **Shipped: phases 01–92** (`.agents/phases/complete/` — read-only
  history; the shipped-features recap of the previous plan revision and
  the README cover it). Migrations through 0015.
- **Next up (`todo/`, in order — already authored via
  `phase-authoring`):**
  1. `93_theme_semantic_completion` — 8 semantic state colors join the
     Theme-tab palette (17 total, B3 revised) + surface panels behind
     every page head.
  2. `94_ls_tree_drilldown` — `ls` becomes a filesystem-style drill-down
     tree with sync-time `lite` folder summaries (`folder_summaries`
     table, migration 0017) + real-model battery.
  3. `95_read_truncation_cap` — capped `read` (default 128 000 chars
     ≈ 32k tokens, `BOR_READ_MAX_CHARS`), honest truncation notice to
     the LLM + visible "(truncated — showing N of M chars)" marker
     (new optional `tool_result` SSE frame — A15 extended to seven
     event types).
- **Next free phase number: 96.**
- **Post-v1 hooks (deliberately not built):** HNSW index at scale;
  inotify auto-import; more providers (the OpenAI-compatible client is
  the seam); multi-user accounts (the `require_user` split is the seam);
  index-backed `grep`.

**Per-phase completion** (AGENTS.md rules 8/9): unit + integration
green, coverage >90%, the phase's Playwright E2E green in isolation, UI
verified against §7, one atomic `--no-gpg-sign` Conventional Commit,
phase dir moved to `complete/`.

---

## 13. House Conventions (quick catch-up)

- **Env:** every setting is a `BOR_`-prefixed env var (or gitignored
  `.env`; see `.env.example`); `get_settings()` is `lru_cache`d.
  Kill switches follow the `agent_max_rounds` pattern: `0` disables the
  feature, a negative value fails startup **loudly** (validator names
  the field).
- **Fail loud, never half-configured:** missing admin secrets, bad
  extension lists, dim mismatches, unresolvable docs branches — all
  refuse to start or refuse the request with a named reason.
- **Shared marker:** `TRUNCATION_MARKER = "[…truncated…]"`
  (`app/rag/retriever.py`) is the only overflow marker; char budgets are
  the pattern for anything sent to `lite`.
- **Byte-identical contracts are load-bearing:** deflected turns
  (A8), empty prompt sections, the B4 theme no-op, and the phase-33/54
  rewrite-only caching all have tests that assert byte-identity.
- **git is invoked only in `scripts/git_sync.py`** (A11 — stdlib
  subprocess); everything else talks Postgres.
- **One story → one phase → one dedicated Playwright file** (AGENTS.md
  rules 4/9); `.agents/` is tracked; only `.agents/phase-sessions/` and
  `.agents/pipeline.log` are gitignored.
- **Full-history references:** the previous full plan (all revision
  notes, complete API table, column-level data model) is in git —
  `git show dac4a3e:.agents/PLAN.md`; per-phase decisions are in
  `.agents/phases/complete/*/00_phase.md`; owner test methodologies in
  `TOOL_CALLING_TESTING.md`; skills for model testing/KB restore in
  `.agents/skills/`.