brain-of-reese/.agents/PLAN.md

Brain of Reese — Master Plan

Status: v1 complete — phases 01–92 shipped (see §12); .agents/phases/todo/ is empty as of this rebuild (2026-09-10). Rule: Every agent reads this file first. Decisions marked LOCKED in the §2 Anchors table are settled — do not re-litigate them in a phase. A change to a LOCKED anchor requires explicit owner permission and is recorded as a dated revision note under the table — never a silent deviation (AGENTS.md rule 3).

Provenance: This file restores the master plan that lived at .agent/PLAN.md until commit 281f355 (removed from VCS by the owner, 2026-08-31). It is re-landed at .agents/PLAN.md — the path AGENTS.md and the codebase reference throughout (PLAN §… comments) — rebuilt from the original text plus every owner-permitted revision recorded in the completed phases through 92.

Revisions (2026-08-21, owner permission): A7/A8/A9 revised (multi-format ingestion, hybrid FTS+vector retrieval, re-tuned honesty gate); dark tech theme (Phase 08); clickable document viewer (Phase 10); thinking display (Phase 17, owner permission 2026-08-23); follow-the-bottom scroll (Phase 18, owner choice 2026-08-23); shared header (Phase 19); whole-document context (Phase 24 — A7's 24k context cap removed, owner permission 2026-08-24).

Revisions (2026-08-25 → 2026-09-09, owner permission): lite summary model (Phase 30); agent document tools ls/read/grep (Phases 37/45/68/70/71/72); single-admin auth + saved chats + shares (Phases 16/50/51/53/79); any-extension import (Phase 56); docs push (Phase 59); SPA shell (Phase 76); LLM retries (Phase 67); login rate limit (Phase 81); security headers (Phase 82); per-source ignore paths (Phase 89); admin Theme tab (Phases 91/92). See the revision log under §2 and the phase records in .agents/phases/complete/.

---

1. Mission

A knowledge base chatbot that embeds Reese's homelab documentation — git repos, local directories, and uploaded archives registered on the admin Sources page — into a Postgres vector database and lets anyone ask Reese (the bot) questions about them.

Product feel: a chippy, upbeat assistant that is optimistic about the user's ability and radically honest — if retrieval didn't surface anything relevant it says "I haven't done anything like that" and offers alternatives instead of hallucinating.

Delivered (v1, phases 01–92)

• Chat UI — mobile-friendly, WCAG 2.1 AA, no CDN — as a single-page shell with deep-linkable views: Chat /, Sources /sources.html, Git sources /git-sources.html, Tuning /tuning.html, History /history.html, Tokens /tokens.html, Theme /theme.html, plus Login, Shared chat, and Document viewer/edit pages.
• RAG over text knowledge files — the A9 family by default (md, markdown, txt, yaml, yml, json, py + the Podman quadlet family + j2; BOR_IMPORT_EXTENSIONS may name any well-formed extension, A9 revised) — from admin-registered git sources, local directory sources, and uploaded archives.
• Hybrid retrieval (pgvector cosine ∪ Postgres FTS, RRF-fused), an honesty gate with honest deflection, whole-document context (never truncated), lite-model summaries of non-markdown docs, a lite-generated KB overview, and grounded-turn agent tools (ls/read/grep).
• Self-hosted models via https://aipi.reeseapps.com/v1 — turbo (chat, with streamed thinking), embed (embeddings, 768 dims — verified), lite (one-shot: document summaries, KB overview).
• Single-admin signed-cookie auth + admin-issued access tokens; saved chats with anonymous share links and KB-generation staleness; steering notes; save-answer-as-doc git push; per-source ignore paths; admin Theme tab; deploy-safe asset caching; security headers; login rate limit.
• Idempotent delta import (uv run python -m scripts.import_docs), the one-click UI Sync sources button, and ample server logging + explicit UI feedback (never a stale submit button).

Still out of scope (deliberately)

• Binary / non-text content ingestion.
• Real-time document watching — the import script / Sync button is the refresh loop (commit → re-run).
• PR tooling for docs push — the owner opens the PR themselves.
• Multi-admin / per-user accounts — one admin + hand-out tokens is the access model.

---

2. Architectural Anchors (LOCKED DECISIONS)

#	Component	Decision	Rationale	Status
A1	Runtime	Python 3.12+, uv for all package management	Fast, reproducible envs; one language for API + tooling	LOCKED
A2	Web framework	FastAPI + Pydantic v2 + Uvicorn	Async, typed, SSE-friendly for LLM streaming, free OpenAPI docs	LOCKED
A3	Database	PostgreSQL 17 (docker.io/postgres:17, pgvector compiled in via db/Containerfile) with cosine (<=>) search	One system for relational + vectors; pgvector is mature; official base image kept per project standard	LOCKED
A4	Orchestration	compose.yaml, started with podman compose up -d	Matches Reese's toolchain	LOCKED
A5	LLM backend	OpenAI-compatible https://aipi.reeseapps.com/v1; models turbo (chat), embed (embeddings), lite (one-shot — A5 extended, phase 30); openai async client	Self-hosted, offline from cloud; no new model management	LOCKED (extended phase 30)
A6	Embedding dim	768 (verified 2026-08-21 against live endpoint via scripts/llm_probe.py); configured by BOR_EMBEDDING_DIM	User recalled 768 — probe confirmed; dimension is fixed at table creation, so mismatch must fail loudly at import time	LOCKED
A7	Retrieval→context	Hybrid: cosine top-100 + Postgres FTS top-30 (OR tsquery, ts_rank) fused with RRF (k=60) → map to parent documents ranked by best fused chunk score → feed the full text of top-N=2 documents (deduped) to the LLM — never truncated (A7 revised 2026-08-24; vector window widened 30→100 on 2026-08-22; summary chunks are first-class retrieval targets, phase 30)	Owner permission 2026-08-21: pure-cosine top-4 missed real docs (gitlab case — best chunk ranked 7th behind vendored-cache junk; score compression 0.41–0.84); the lexical signal finds name-your-tool questions; the 100-wide vector window lets double-hit chunks outrank generic templates (measured on the live 2774-chunk KB, 2026-08-22); whole-document context contract preserved — no […truncated…] cap (owner: "this should never happen"; emergency-valve variant rejected)	LOCKED (revised 2026-08-24)
A8	Honesty gate	Deflection mode (LLM must NOT pretend to know; offers 2–3 alternative questions about things it DOES have notes on) when best cosine < BOR_RELEVANCE_THRESHOLD and no candidate chunk FTS-matches the question; threshold re-tuned for the embed model's compressed score range (default 0.62, calibrated via scripts/eval_retrieval.py; the E2E mock uses its own 0.30 calibration via the app fixture). Deflected prompt: weak-hit titles only (never document content) + the DEFLECT_MODE marker + the phase-71 plain-text line "Reply in plain text only — you have no tools in this mode."	Owner permission 2026-08-21: at 0.30 the gate never discriminated (measured corpus range 0.41–0.84); the FTS-OR keeps name-your-tool questions honest-positive; deflection product behavior unchanged. Phase 71 (owner-permitted 2026-09-03): the one added line closes the tool-markup door at the prompt — the DEFLECT_MODE marker contract is unchanged	LOCKED (revised 2026-08-21)
A9	Content scope	Text formats md, markdown, txt, yaml, yml, json, py + the Podman quadlet family (container, network, volume, image, pod, kube, swap, os, endpoint) + j2 (default, BOR_IMPORT_EXTENSIONS — may name any well-formed extension or narrow to a subset, A9 revised 2026-08-31), hidden (dot) directories/files skipped, plus the exclusion list (node_modules, __pycache__, .pytest_cache, dist, build, …), plus per-source ignore paths (phase 89: raw source-relative prefixes, no globs, git_sources.ignore_paths)	Owner permission 2026-08-21: real notes live in yaml/py/json/txt too; the dot-dir skip removes the ~470 vendored-cache junk docs (.esphome/.espressif/**, …) that outranked real content. Phase 47 (2026-08-27): quadlet + j2 join the default — plain-text chunking. Phase 56 (2026-08-31): any well-formed extension allowed (the A9 family is a default, not a ceiling); a blank list or malformed token fails startup loudly, naming the value	LOCKED (revised 2026-08-31)
A10	Auth	None in v1; all endpoints stateless under /api	Per user (auth later); statelessness keeps the future migration cheap	LOCKED (revised phase 16 — see revision log)
A11	Frontend	Vanilla HTML/CSS/JS in git; no CDN — everything served by FastAPI StaticFiles; minified by esbuild in the Containerfile build stage; system font stack	No external deps at runtime; tiny, auditable surface; mobile-friendly by construction	LOCKED (extended phase 76 — see revision log)
A12	Aux services	None (no Valkey, no SeaweedFS, no queue) — everything in-process: session state is a signed cookie (no store), the Sync button runs the import in-process, the login rate limit is in-memory per-process	No server-side session state; a homelab single-box deploy; add later only if a need appears	LOCKED
A13	Migrations	Alembic + SQLAlchemy 2.0 (sync) + psycopg 3; every migration ships a tested downgrade	Standard, reversible, reviewable schema history	LOCKED
A14	Debugging	debugpy only when DEBUGPY=1 (env var read directly, not via settings); listen 0.0.0.0:5678 (override DEBUGPY_PORT), non-blocking, attach-on-demand; not imported at all when off	Zero overhead by default per project standard; attach-on-demand keeps production runs clean	LOCKED
A15	Chat transport	SSE streaming from POST /api/chat — event types thinking, delta, tool, retry, done, error (§4)	Local LLM latency is 10–30s; live token stream + explicit completion event power the UI's feedback states	LOCKED (extended phases 17/37/45/67 — see revision log)
A16	Testing	Per phase: unit + integration (pytest, coverage >90% on app/) + one dedicated Playwright E2E file per user story, run in isolation; E2E uses a deterministic mock LLM by default (E2E_REAL_LLM=1 opts into live aipi)	One story, one phase, one E2E gate — the pipeline's core invariant	LOCKED
A17	Git	Conventional Commits, always --no-gpg-sign, repo-local commit.gpgsign=false; one atomic commit per completed phase	Subsequent agents may lack the GPG key	LOCKED
A18	Docs push	Save a chat answer as documentation: server-side draft (doc_drafts, the long body never lives in a URL — an unguessable uuid4 token is the edit-screen URL credential), pushed to a generic git remote (BOR_DOCS_REPO — URL or local path; no gh, no GitHub assumption) on a dedicated branch (git push --ff-only, re-cut from BOR_DOCS_BASE_BRANCH per push); no PR tooling — the owner opens the PR; the feature is inert (hidden + 409) while BOR_DOCS_REPO is empty	Phase 59, owner permission 2026-08-30 (decisions D1–D3): answers should flow back into the docs with a review step, without coupling the app to one forge; the empty-repo kill switch mirrors the git-sources env-fallback pattern	LOCKED (added phase 59)
A19	Deploys & caching	One Starlette middleware: HTML pages ship Cache-Control: no-cache (no etag/last-modified); assets are versioned (/assets/*?v=<token>) and ship Cache-Control: public, max-age=31536000, immutable; the token is the deploy — the short SHA of git HEAD (content hash of the frontend/ tree without a .git)	A deploy is a commit — the browser must see it without a hard refresh; no CDN, no new services, no build-step change (phases 33 + 54)	LOCKED (added phase 33, revised phase 54)
A20	Security headers	Every response carries Content-Security-Policy: default-src 'self' (the themed HTML pages additionally allow the inline <style> hash of the current theme), frame-ancestors 'none', and X-Content-Type-Options: nosniff; login is rate-limited in-memory (10 attempts / 15 min, stdlib-only, per-process — A12)	Owner-approved policy, phase 82 (audit SEC findings); the CSP hash keeps the theme's pre-paint <style> legal without unsafe-inline anywhere else	LOCKED (added phase 82)

Revision log (owner permission noted — the established style)

A10 revision (phase 16, owner permission 2026-08-22): single-admin signed-cookie auth — public: health (and, until the phase-79 split below, chat / documents / suggestions); admin-only: docs catalog + steering. The row above keeps the original v1 decision text; the public API surface stays stateless (the signed session cookie is the only session state) — recorded as a revision, not a silent deviation. Fail-loud at boot while BOR_ADMIN_PASSWORD or BOR_SESSION_SECRET is empty.

A10 UI revision (phase 19, owner permission 2026-08-23): the "Sources" nav link is hidden from anonymous users on all pages — the soft-gate page and the API split above are unchanged.

A10 extension (phase 50, owner permission 2026-08-29): conversations the owner explicitly Saves persist in saved_chats (one row per saved conversation, the bor.chat.v1 message list as JSONB); POST /api/chat itself stays stateless and nothing is stored about a conversation that was not saved.

A10 extension (phase 51, owner permission 2026-08-30): a saved chat can be shared anonymously — share_token (NULL = private, uuid4 = publicly readable at /shared/<token> without any session); unsharing revokes it. Shared chats are the only content that stays anonymous.

A10 extension (phase 53): saved chats stamp the KB generation (sources_version, bumped once per KB-changing sync in sources_meta) and are surfaced as stale (regenerable) when older than current.

A10 extension (phase 74, owner-locked 2026-09-08): POST /api/chat accepts an optional client-provided history (the prior turns, stateless per A10 — nothing is stored): kept while both budgets hold (BOR_HISTORY_MAX_TURNS=40 turns, BOR_HISTORY_MAX_CHARS=24000 chars counting prior thinking), trimmed newest-first, a turn dropped WHOLE never cut mid-answer; who="brain" turns map to assistant messages with reasoning_content only when thinking is non-empty (the preserve-thinking wire convention).

A10 extension (phase 79, owner permission 2026-09-08): admin-issued access tokens (bor_ + 32 hex) — the only stored credential is the SHA-256 digest of the full token (shown once at create); a token user can chat and open cited documents, nothing else (chat, suggestions, documents; the admin surfaces stay 403); revocation is immediate (live-checked on the holder's next request).

A11 extension (phase 76, owner permission 2026-09-08): the navbar views (Tuning, Sources, Git sources, History, Tokens, Theme) are views of ONE shell document (frontend/index.html) — the client-side router (frontend/assets/router.js) shows the view matching location.pathname, so direct URLs deep-link; the shell routes are registered after the API routers and before the static catch-all, so the A19 caching contract applies untouched.

A15 extension (phase 17, owner permission 2026-08-23): the contract gains {"type":"thinking","text":"…"} — the model's reasoning ahead of the delta events (turbo emits delta.reasoning_content chunks before the first content chunk, verified live 2026-08-23); BOR_STREAM_THINKING=0 suppresses the frames server-side (the chars are still counted for the §9 line). delta and done shapes unchanged.

A15 extension (phase 37, owner permission 2026-08-26): the contract gains {"type":"tool","name":"…","argument":…|null} — the model's document tool calls on grounded turns. Deflected turns keep the direct stream — the LOW prompt never carries tools.

A15 revision (phase 45, owner permission 2026-08-27): the phase-37 per-tool budgets (BOR_AGENT_LIST_CALLS / BOR_AGENT_READ_CALLS) are removed (owner: "allow the LLM to make as many tool calls as it wants" — TODO.md L8); BOR_AGENT_MAX_ROUNDS (default 10) caps the tool rounds and 0 disables the tools entirely (the pre-phase-37 path — one tools=None request, the kill switch). The tool and done shapes are unchanged; done.sources includes the agent-read documents (deduped by (source, path), order preserved).

A15 extension (phase 67, owner-locked 2026-09-01): the contract gains {"type":"retry","attempt":n,"max_attempts":N} — when the aipi endpoint dies before a request has streamed its first output frame, the request restarts up to BOR_LLM_RETRIES (default 3) times, a flat BOR_LLM_RETRY_DELAY (default 5 s) between attempts; every restart is announced with a retry frame so the UI shows the transient "Communication interrupted — retrying (n of N)…" status, not an error. Exhaustion and any failure after the first frame keep the terminal error frames (byte-identical). The pre-stream question embedding retries on the same budget.

A15 revision (phase 70, owner permission 2026-09-03): the tool surface is the harness-trained ls / read / grep (replacing phase 37's list_documents / read_document, plus the grep locator of phase 68): ls lists indexed documents as source: X | path: Y | title: Z lines (optional path = a source name, not a directory); read takes the combined source/path string including the source name and appends the full document, never truncated; grep is a case-insensitive fixed-substring locator (up to 20 source/path:line: text matches, optional one-document scope) — a locator that adds no source or context (the model reads the winner). argument in the tool frame is the single string the model passed (read's path, grep's pattern, ls's path) or null.

A12 reaffirmed (phases 32/64/81): the Sync button and the model liveness probe run the work in-process (no queue, no new service); the login rate limit is in-memory per-process stdlib (a restart clears the counter — accepted).

---

3. High-Level Architecture

                       ┌────────────────────────────────────────────┐
                       │              Podman Compose                │
   Browser             │  ┌──────────────────────────────────────┐  │
   ┌──────────┐  HTTP  │  │  brain-of-reese/app (FastAPI)        │  │
   │ index.html│◄──────┼─►│  • SPA shell + views (no CDN, A11)   │  │
   │ app.js    │  SSE   │  │  • SessionMiddleware (signed cookie)│  │
   └──────────┘        │  │  • SecurityHeaders + caching middle-│  │
                       │  │    ware (A19/A20)                   │  │
                       │  │  • /api/* (chat, docs, git-sources,  │  │
                       │  │    steering, sync, chats, tokens, …)│  │
                       │  │  • RAG pipeline (app/rag/)          │  │
                       │  │    embed→retrieve→gate→agent→turbo  │  │
                       │  └──────┬──────────────────┬───────────┘  │
                       │         │ SQL (psycopg)    │ OpenAI-compat│
                       │  ┌──────▼──────┐   ┌───────▼────────────┐ │
                       │  │ db:         │   └─────────┬──────────┘ │
                       │  │ postgres:17 │             │            │
                       │  │ + pgvector  │             │            │
                       │  └─────────────┘             │            │
                       └──────────────────────────────┼────────────┘
                                                      ▼
                                       https://aipi.reeseapps.com/v1
                                       (self-hosted: turbo, embed, lite)

   Source registry (Postgres `git_sources`, admin-managed):
     git repos  → cloned/pulled into BOR_SOURCES_DIR (git CLI via
                  scripts/git_sync.py — the ONLY place git is invoked)
     local dirs → walked directly (no checkout)
     archives   → unpacked under BOR_UPLOAD_DIR (zip-bomb guarded)

   Offline tooling (same repo, same venv):
     scripts/import_docs.py    → walks the sources, chunks, embeds, upserts
     scripts/eval_retrieval.py → ranks hybrid results for a question (tuning)
     scripts/llm_probe.py      → verifies models + embedding dim
     scripts/model_benchmark.py / test_embed_model.py /
     test_summary_model.py     → model benchmarks (benchmarks/)

Component breakdown

Component	Responsibility	Lives in
App (FastAPI)	Serves the SPA shell + /api; session auth; middlewares (caching A19, security headers A20)	app/main.py, app/core/
RAG pipeline	embed → hybrid retrieve (cosine ∪ FTS, RRF) → honesty gate → prompt assembly (persona + KB overview + tuning + documents + tools) → turbo streamed, with the agent loop for grounded turns	app/rag/ (retriever, prompts, agent, llm, scaffolding, chunker, importer, summarizer, overview, git_sources, archive_upload, source_removal, sources_meta, suggestions)
Importer	Source walk (A9 scope, ignore paths), sha256 delta, format-aware chunking, batched embedding, two-phase upsert/prune, lite summaries, KB overview regeneration, sources-version bump	app/rag/importer.py + scripts/import_docs.py + app/api/sync.py (UI Sync)
Auth & access	Fail-loud admin config; signed cookie (require_admin / require_user); token auth + admin token API; login rate limit	app/core/auth.py, app/core/tokens.py, app/core/rate_limit.py, app/api/auth.py, app/api/tokens.py
DB	11 tables + vector extension (migrations 0001–0015, each reversible)	db/ image, alembic/versions/
Frontend	SPA shell + views (chat, sources, git-sources, tuning, history, tokens, theme, login, shared, document viewer), feedback state machine, theming	frontend/ (index.html shell + document.html / shared.html / login.html / doc-edit.html)

Chat data flow (LOCKED A7/A15)

user question (+ optional client history)
  → POST /api/chat {message, history?}
  → embed(question)                    [aipi /v1/embeddings, model=embed; retried]
  → cosine top-100 ∪ FTS top-30 (OR tsquery, ts_rank)   [pgvector + PG FTS]
  → RRF fuse (k=60) → docs ranked by best fused chunk score
  ├─ best cosine >= BOR_RELEVANCE_THRESHOLD (0.62) OR fts_hits > 0
  │    → HIGH: locked persona + <knowledge_base> + <tuning> + full top-2
  │      documents + <tools>  →  agent loop (ls/read/grep, round-capped)
  │      → turbo streamed: thinking → tool → delta events → done
  └─ else → LOW: DEFLECT_MODE prompt (weak titles only + plain-text line)
            → turbo streamed (no tools, byte-identical path) → done
  → query_log row + the per-turn log line (§9)
  → final "done": {deflected, sources[] (retrieval + agent-read, deduped),
                   suggestions[]}

---

4. API Design

Errors: standard JSON {detail: str}. Auth: require_admin (the admin session), require_user (admin or a live access token), or public (no dependency) — per the A10 revisions.

Method	Path	Auth	Purpose
GET	/api/health	public	Liveness + db up/down + version
GET	/api/config	public	UI identity: app name, placeholder, footer text (effective ui_settings over env)
POST	/api/login	public (rate-limited)	Password → signed bor_session cookie (204)
POST	/api/logout	public	Clears the session cookie (204)
GET	/api/whoami	public	{authenticated, role: admin|user|anonymous}
POST	/api/token-auth	public	Access token (bor_…) → same cookie, role user (204)
GET	/api/suggestions	user	Onboarding chips: BOR_SUGGESTIONS seed until any question is saved, then the last 3 questions asked
GET	/api/docs	admin	Indexed document list (source, path, title, chunks, indexed_at)
GET	/api/documents/content?source=…&path=…	user	One indexed document's full content (feeds the viewer)
PATCH	/api/documents/summary	admin	Re-generate/edit one document's lite summary (phase 57)
POST	/api/chat	user	RAG chat turn → SSE stream
GET/POST/PUT/DELETE	/api/steering[/<id>]	admin	Steering notes (1–2000 chars each; injected as <tuning>)
GET	/api/git-sources	admin	Source registry rows (kind: git|local, ignore paths)
POST	/api/git-sources	admin	Add a git URL or local directory (201)
DELETE	/api/git-sources/<id>	admin	Remove a source (re-importable; never touches the KB until a sync)
POST	/api/git-sources/upload	admin	Upload an archive source (202 → unpack + register; the scan is deferred to Sync — phase 90)
GET	/api/git-sources/upload/status	admin	Upload progress
POST	/api/sync	admin	Run the full source refresh in-process (202; 409 while one is running)
GET	/api/sync/status	admin	Sync progress (poll every 2 s; no client timeout)
GET/POST/PUT/DELETE	/api/chats…	admin	Saved chats (list / save / update / delete / POST /api/chats/<id>/share)
GET	/api/shared/<token>	anonymous	One shared chat's JSON snapshot
GET	/shared/<token>	anonymous	The shared-chat page
POST/GET/PUT	/api/doc-drafts[/<token>]	admin	Save an answer as a doc draft (201); load/edit by token
POST	/api/doc-drafts/<token>/push	admin	Commit + push to BOR_DOCS_REPO (A18; 409 while the repo is unconfigured)
GET/POST	/api/tokens, POST /api/tokens/<id>/revoke	admin	Access-token surface (the token string appears once, at create)
GET/PUT	/api/ui-settings	admin	The single ui_settings row (Theme tab; NULL = default, B1)

Phase 79: chat, suggestions, and cited documents moved behind require_user (admin or a live access token) — the only fully anonymous content is the shared-chat read above.

SSE contract (POST /api/chat)

data: {"type":"thinking","text":"…"}\n\n        ← model reasoning (before the first delta)
data: {"type":"tool","name":"ls","argument":null}\n\n   ← grounded turns only, one per model call
data: {"type":"retry","attempt":1,"max_attempts":3}\n\n ← endpoint died pre-first-frame
data: {"type":"delta","text":"Hey! "}\n\n         ← answer tokens
...
data: {"type":"done","deflected":false,
       "sources":[{"source":"Homelab","path":"kubernetes.md","title":"…"}],
       "suggestions":[]}\n\n

and on failure: data: {"type":"error","detail":"…"}\n\n (terminal — no done, no query_log row; a pre-stream DB outage is a plain 503 JSON instead).

Client rules:

• Render deltas live; render thinking in the collapsible block above the answer (auto-collapses on the first delta, toggleable after, persisted with the message).
• Render each tool frame as a transient "calling tool" status (the send button stays busy — "Stop" — for the whole turn; Stop closes the connection, which cancels the server generator).
• retry frames show the transient "Communication interrupted — retrying (n of N)…" status — never an error banner.
• On done: append source chips (clickable → document viewer) / suggestion chips, clear the busy state.
• Guards: a 300 s pre-token guard (TURN_TIMEOUT_MS) turns a hung stream into the error state; a button can never sit "stuck" forever (the §7.4 contract).

---

5. Data Model (PostgreSQL 17)

Migrations alembic/versions/0001…0015 (idempotent CREATE EXTENSION IF NOT EXISTS vector); every migration's downgrade is tested (A13). Single-row tables use id = 1 (the kb_overview precedent).

documents

Column	Type	Notes
id	UUID PK
source	VARCHAR(120)	source name (dir basename / repo name / archive name)
path	VARCHAR(1000)	relative to the source root
full_path	VARCHAR(2000)	absolute path at import time (diagnostics)
title	VARCHAR(500)	first markdown H1, else file stem
content	TEXT	full text — the RAG context
content_hash	VARCHAR(64)	sha256 of content — change detection
indexed_at	TIMESTAMPTZ
summary	TEXT NULL	lite-model summary, non-markdown docs only (phase 30) — NULL on markdown docs and on the best-effort failure path
—	UNIQUE (source, path)	upsert key

chunks

Column	Type	Notes
id	UUID PK
document_id	UUID FK→documents CASCADE	embedding→document mapping
position	INT	0-based order; −1 = the summary chunk
content	TEXT	chunk text (format-aware)
embedding	VECTOR(768)	nullable until embedded (two-phase import)
is_summary	BOOL default false	marks the position −1 summary chunk (phase 30)
tsv	TSVECTOR	generated to_tsvector('english', content) STORED + GIN index (A7)

No vector index in v1: sequential scan is fine at this corpus size. Revisit with an HNSW index if retrieval latency grows (§13).

query_log

id UUID PK, question TEXT, top_score FLOAT, fts_hits INT, chunk_hits INT, deflected BOOL, sources TEXT (comma-joined source/path, retrieval + agent-read deduped), latency_ms INT, created_at TIMESTAMPTZ — the durable record behind threshold tuning.

steering_notes (phase 15)

id UUID PK, note TEXT (1–2000 chars, API-enforced), created_at — injected oldest-first into every turn's system prompt as <tuning>, char-budgeted (BOR_STEERING_MAX_CHARS).

kb_overview (phase 31)

id INT PK (=1), content TEXT, updated_at — single-row lite-generated outline of the KB, regenerated when an import changed the KB; injected as <knowledge_base> (one indexed PK read per turn — no LLM call).

git_sources (phases 35/38/89)

id UUID PK, url TEXT unique, kind TEXT ('git'|'local' CHECK), path TEXT unique NULL (local only), ignore_paths JSONB default '[]' (phase 89 — normalized source-relative prefixes), added_at — the admin-managed source registry; DB rows are the source of truth, BOR_GIT_SOURCES (git-only) is the empty-table fallback.

saved_chats (phases 50/51/53)

id UUID PK, title VARCHAR(500) (auto: first question), messages JSONB (the exact bor.chat.v1 record — raw text, never HTML), share_token UUID unique NULL (phase 51 — NULL = private), sources_version INT default 0 (phase 53 — the KB generation at save), created_at, updated_at.

sources_meta (phase 53)

id INT PK (=1), version INT (KB generation), updated_at — bumped exactly once per sync that changed the KB (added + updated + pruned > 0); a saved_chats row older than it is stale.

doc_drafts (phase 59)

id UUID PK, token UUID unique (the URL credential — never the row id), title TEXT, path TEXT (in-repo file path, default docs/<slug>.md), body TEXT (the raw markdown answer), status TEXT ('draft'→'pushed'), branch TEXT NULL, commit_sha TEXT NULL, created_at, updated_at.

api_tokens (phase 79)

id UUID PK, label VARCHAR(120) (display-only), token_hash VARCHAR(64) unique (SHA-256 of the FULL bor_… string — the plaintext exists only in the 201 create response, shown once), created_at, last_used_at NULL, revoked_at NULL (set = dead, live-checked).

ui_settings (phase 91)

id INT PK (=1), app_name / input_placeholder / footer_text VARCHAR(300) NULL, bg / surface / ink / ink_soft / line / grid_line / brand / brand_soft / brand_ink VARCHAR(7) NULL — the Theme tab's single row; B1 (owner-locked 2026-09-09): every column NULL/empty = "use the default" (the env value for the strings — BOR_APP_NAME etc. — and the built-in palette for the colors; colors have no env fallback).

Document state transitions

unseen ──import──▶ indexed ──hash changed + re-import──▶ reindexed
                     │
                     └──file deleted/--prune/ignored──▶ removed (chunks cascade)

Chunking policy (format-aware — app/rag/chunker.py)

• Markdown: split on ## /### headings; sub-split sections over BOR_CHUNK_TARGET_CHARS (2000) at paragraph boundaries with BOR_CHUNK_OVERLAP_CHARS (200) overlap; each chunk keeps its nearest preceding heading.
• YAML/YML: top-level keys + --- separators (key line kept as anchor); JSON: pretty-printed, top-level keys; Python: top-level defs/classes (stdlib ast); txt: paragraphs.
• Quadlet family + j2: plain-text chunking — no format-specific splitter (A9 revised 2026-08-27).
• Every format honors the 1200-char hard cap (aipi ~1024-token request limit; ~1.4 chars/token) — oversized blocks split by line.

---

6. RAG Pipeline & Persona

Locked system prompt (sent with every chat turn — app/rag/prompts.py)

You are "Brain of Reese" — the digital brain of Reese, a self-hoster and
homelab tinkerer. Personality: chippy, upbeat, warm, and genuinely
optimistic about the user's ability to do things.

Rules:
1. Answer ONLY from the provided document context. Cite which document(s)
   you used, by path.
2. Be concrete: names, versions, ports, hosts, schedules — the specifics in
   the docs are the value.
3. HONESTY GATE: if <relevance> is "LOW", you must NOT pretend to know.
   Offer 2-3 alternative questions about things you DO have notes on.
4. Never invent facts, hosts, or steps that are not in the context.
5. Keep answers tight: short paragraphs, bullets where helpful.

<relevance>{HIGH|LOW}</relevance>

Change this text through the plan, not in code (phase 03 convention).

§6 revision (2026-08-22, owner permission): the owner's working-tree persona edits are preserved — no mandated "you've got this" tagline and no mandated deflection opening; the honesty gate itself is unchanged.

Prompt section order (both modes): <relevance> → <knowledge_base> (when the kb_overview row is non-empty) → <tuning> (when steering notes exist) → mode body. An empty section omits itself — a no-notes/no-overview prompt is byte-identical to the pre-steering text (phase 15 convention, re-used by phase 31). Both sections are char-budgeted (BOR_KB_OVERVIEW_MAX_CHARS / BOR_STEERING_MAX_CHARS) with the shared […truncated…] marker.

• HIGH mode appends the full top-N document texts under <documents> as <document source="…" path="…" title="…"> blocks, then the <tools> section (below) — the instructions are the last thing the model reads.
• LOW mode (deflection) appends the DEFLECT_MODE marker + the weak-hit titles only (never document content) + the phase-71 plain-text line "Reply in plain text only — you have no tools in this mode." The DEFLECT_MODE marker's presence is the E2E mock's key — wording may not change without the mock.

The <tools> section (HIGH only — LOCKED, phase 37/70/71/72)

Teaches the harness-aligned shapes: ls (lists indexed documents as source: X | path: Y | title: Z; optional path = a source name, not a directory); read (combined source/path including the source name — a bare path will not resolve; do not call read for a document already in <documents> — answer from the text in the prompt); grep (exact case-insensitive substring, up to 20 source/path:line: text matches — a locator, not a context-adder; plain substring, NEVER a regex; for a normal pass only pattern). Discipline rules (phase 72, from live gate telemetry): exactly one tool call per reply (a batched second call is discarded); never repeat a refused or already-succeeded call; path arguments are document identities, never source names. The round cap is enforced by the agent loop, not re-stated in the prompt (phase 45).

Agent loop (grounded turns — app/rag/agent.py)

A grounded turn runs run_agent: the model may call the tools as many times as it wants until it answers or BOR_AGENT_MAX_ROUNDS (default 10) forces one final no-tools answer. Each executed call streams a tool frame; rejected calls (bad identity shapes) get the teaching refusals (phase 72 — deterministic, model-free) instead of execution. Tool execution is Postgres-only (no extra LLM round trip). Deflected turns never run the agent — the direct chat_stream path is byte-identical (A8). Raw chat-template scaffolding tokens (<|tool_call_start|>…) are filtered deterministically (app/rag/scaffolding.py, phase 71) with exactly one bounded recovery; a wipe that survives recovery settles with the structured error frame.

Retrieval (hybrid — A7/A8)

• Embed the question (embed, 768-d) → cosine top-100 candidates.
• Lexical: OR tsquery over the question's tokens → FTS top-30 by ts_rank (a stored tsvector, GIN-indexed).
• Reciprocal Rank Fusion (Σ 1/(k+rank), k=60) → distinct parent docs ranked by best chunk's fused score → top BOR_TOP_N_DOCS (2) → full content — never truncated. Summary chunks (position −1) resolve to the parent document like any chunk; the per-turn line reports summary_hits=.
• Honesty gate: LOW only when best cosine < BOR_RELEVANCE_THRESHOLD (default 0.62 — the embed model's live cosines cluster ~0.6–0.85) and zero FTS hits among the candidates.

---

7. UI/UX Strategy

7.1 Layout structure

• App frame: sticky header (64px, --header-h) + <main> (flex-grow)
  • footer. Container: max-width: 72rem; margin-inline: auto; padding-inline: 1.25rem. body { min-height: 100dvh } — no fixed body height (it pins the sticky navbar's travel range, phase 60).
• SPA shell (phase 76): the navbar views are views of ONE shell document — the router shows the view matching location.pathname; direct URLs deep-link. Switching views keeps a live stream alive where the contract requires (phase 73: hidden-tab streaming).
• Chat: a centered reading column — --chat-column: 46rem base, 2× (92rem) at ≥1500px wide desktops (owner instruction 2026-08-31, phase 58 — chat + shared + document view). Deliberate: chat is a vertical conversation — a centered, capped column is the correct pattern (NOT a layout bug); the 72rem frame + header/footer keep it from reading as a hairline.
• Sources page: full-width responsive table (min 640px, horizontal scroll wrapper on small screens) + stat cards in grid-template-columns: repeat(auto-fit, minmax(170px, 1fr)). No skinny single-column lists anywhere: lists/tables/grids use ≥80–90% of the container width.
• Document viewer: near-fullscreen modal on the current page (#doc-modal) — same-page, no new tab; back keeps the page state.
• Mobile (≤640px): hamburger nav; suggestion chips horizontally scrollable; composer stays reachable with safe-area-inset-bottom; touch targets ≥44px.

7.2 Accessibility (WCAG 2.1 AA) & theming

• Semantic landmarks on every view: <header>, <nav aria-label>, <main>, <footer>; skip-link to #main.
• Every control labeled: visible <label> or aria-label (icon-only buttons always get aria-label); form inputs have (visually-hidden) labels.
• Live regions: message stream aria-live="polite"; typing indicator role="status"; banners role="status"; errors role="alert".
• :focus-visible outline 3px; prefers-reduced-motion respected by all animations and stills the background texture.
• Theming (phases 08 → 91/92): the built-in dark palette (app/core/theming.py::BUILTIN_COLORS — bg #0f0a0a, surface #1a0f0f, ink #f0e6e6, ink-soft #b8a8a8, line #2d1a1a, grid-line #4a2626, brand #f43f5e, brand-soft #2d0a0a, brand-ink #fca5a5) renders byte-identically when nothing is set. Text-on-surface pairs verify ≥4.5:1 (ink/surface ≈15.3:1, ink-soft/surface ≈8.2:1, brand-ink/brand-soft ≈9.5:1, dark-on-brand ≈5.4:1; light ink on brand ≈3.0:1 — never used for text). The admin Theme tab (/theme.html, phase 91) edits the 9 identity colors + the 3 identity strings, persists them in ui_settings (B1: NULL = default), and injects them as an inline <style> before first paint (no red flash) — the retired CSS-file BOR_THEME env is ignored; the page's CSP carries the theme's hash (A20).

7.3 No external dependencies

• System font stack only (no font files to bundle, no CDN fonts).
• Zero <script src="https://…"> / <link href="https://…"> — enforced by an integration test on the index page and re-checked by every UI phase's verification step (AGENTS.md rule 6).
• Markdown rendering is a small local function (escape-first, then transform) — XSS-safe, no library.
• Container build minifies the frontend with esbuild (build-time only).

7.4 Visual feedback standard (the "never stale" contract)

State	UI
Idle	Send button enabled, label "Send".
Thinking (pre-token)	3-dot typing bubble + button busy, label "Thinking…"; elapsed hint after a few seconds.
Thinking (model reasoning)	Collapsible .thinking block streams open (the live indicator), auto-collapses on the first answer token, toggleable afterwards, persisted with the message (phase 14); it scrolls internally and never drags the page (phases 21/43).
Calling a tool	Transient "calling tool" status line (same pattern as the thinking indicator); the button stays busy — "Stop" — for the whole turn (phase 37; the read line shows progress for big reads, phase 87).
Streaming	Deltas append live into the brain bubble; button stays busy.
Retrying	retry frames → transient "Communication interrupted — retrying (n of N)…" status — never an error banner (phase 67).
Done (answer)	Source chips under the bubble (mono, path-based, clickable → viewer); button re-enabled; answer actions (Tune / Save / Copy / Retry, phase 65).
Done (deflected)	Amber bubble + "Maybe try:" suggestion chips (closest indexed titles).
Error	Red banner (role="alert") with retry hint; button re-enabled; "Retry answer" re-asks the last question (phase 49).
KB offline	Amber banner at top of chat ("start Postgres…"); chat disabled with explanation.
Stop	"Stop" closes the SSE connection; the partial answer stays, marked stopped; no autoscroll.
Guard	300 s pre-token guard (TURN_TIMEOUT_MS, raised from 120 s) → error state (a button can never sit "stuck" forever). The guard counts only visible time — a hidden tab doesn't burn it (phase 73).
Scroll	No auto-follow during a turn (owner direction 2026-08-27, phase 42 — revising the phase-18 follow-the-bottom choice): the viewport moves only on user intent — submit reveals the user's message; restoring a saved conversation lands one-shot on the latest message.

Every failure path re-enables its controls (the "never stale" rule) — pinned by unit tests on the app.js state machine and by the story E2E suites.

7.5 Component inventory (core ids used by tests — shell era)

Chat: #messages, #empty-state, #suggestions, .suggestion-chip, #composer, #message-input, #send-btn / #send-label, #typing-indicator, .msg.user/.msg.brain .bubble, .source-chip, .msg.brain.is-deflected, .thinking / .thinking-text, #kb-banner, #app-version, #doc-modal (the viewer modal). Shell/nav: #app-nav (+ the hamburger on mobile), #main. Sources: #stat-docs, #stat-chunks, #stat-last, #docs-table, #docs-tbody, #sources-empty, the Sync button. Git sources: #git-sources-table, #git-source-form, #archive-upload-form, #ignore-editor-dialog. History: #history-tbody, #history-gate. Tokens: #tokens-table. Theme: #theme-form (the 9 color fields + 3 string fields). Login gate: #auth-gate (chat) / #sources-gate / #history-gate (soft gates).

---

8. Debugging (debugpy protocol)

• DEBUGPY unset/0 → debugpy is never imported (verified by unit test — A14).
• DEBUGPY=1 → listener on 0.0.0.0:${DEBUGPY_PORT:-5678}, non-blocking, app continues; IDE attaches on demand.
• Entry point: app/core/debugging.py::configure_debugging() called at the top of app/main.py module import — so uv run uvicorn app.main:app, python -m scripts.…, and tests all honor it.
• VS Code: "type": "debugpy", "request": "attach", "connect": {"host": "localhost", "port": 5678}.

---

9. Observability

• App logs: single-line timestamp LEVEL logger :: message on stdout; INFO by default (BOR_LOG_LEVEL); noisy third-party loggers capped at WARNING. Every request-critical operation logs key=value context at INFO — this pairs with the UI feedback contract (§7.4).
• Per-chat-turn log line (required — one per completed turn):

  question=… embed_ms=… top_score=… fts_hits=… summary_hits=… tuning=N
  kb_chars=N history_msgs=N threshold=… deflected=… sources=…
  thinking_chars=… tool_calls=N total_ms=… retries=N
  scaffold_stripped=N
  

  Field notes: summary_hits= (phase 30) — hit summary chunks whose parent doc is in the context; kb_chars= (phase 31) — the injected overview length; history_msgs= (phase 74) — kept history messages; sources= — retrieval docs + agent-read docs, deduped (empty on deflected turns); thinking_chars= (phase 17) — counted even when BOR_STREAM_THINKING=0; tool_calls= (phase 37) — executed calls (rejected calls don't count); retries= (phase 67) — 0 when nothing was retried; scaffold_stripped= (phase 71) — stripped scaffolding chars across the turn's requests, 0 on clean turns. A cancelled turn writes neither the line nor a query_log row.
• Importer logs: per-file added | updated | unchanged | pruned + per-summary import: summary source=… path=… chars=… + the greppable summary line (cron-safe):

  import: summary files=… added=… updated=… unchanged=… pruned=…
  errors=… chunks=… embed_batches=… summaries=… summary_errors=…
  formats=md:203,yaml:267,py:14     (highest count first)
  

• query_log table: durable record of every question (score, FTS hits, deflection, sources, latency) for threshold tuning and doc-gap finding.

---

10. Testing Strategy (LOCKED — A16)

Layer	Tooling	Runs	Gate
Unit	pytest	uv run pytest tests/unit	pass
Integration	pytest + FastAPI TestClient	uv run pytest tests/integration	pass
Coverage	pytest-cov on app/	uv run pytest --cov=app --cov-report=term-missing	>90% per phase
E2E	Playwright (sync API), one file per story	uv run pytest tests/e2e/test_<story>.py -v --no-cov	passes in isolation

• E2E determinism: tests/e2e/mock_llm.py serves a deterministic OpenAI-compatible API. Embeddings are genuine L2-normalized token-overlap vectors, so the cosine threshold behaves like production: on-topic questions retrieve, off-topic questions deflect. tests/e2e/slow_llm.py is the slow/dead-endpoint variant for the retry/timeout suites. E2E_REAL_LLM=1 switches the fixture to live aipi (needs an imported KB).
• E2E prerequisites: podman compose up -d db; Chromium via uv run playwright install chromium.
• DB isolation: story E2E fixtures truncate query_log (and re-import fixtures for import-dependent stories) per test module.
• Lint/types gate: uv run ruff check . and uv run pyright clean.

---

11. Import & Update Workflow (documented in README)

# the whole update loop (git sources): commit → re-run
uv run python -m scripts.import_docs                # sync-clone/pull + walk + index
uv run python -m scripts.import_docs --prune        # also drop deleted/ignored files
uv run python -m scripts.import_docs --source ~/OtherProject   # extra dirs
# or one click: the admin Sources page's "Sync sources" button (in-process)
uv run python -m scripts.eval_retrieval "How did I install gitlab?"
uv run python -m scripts.llm_probe                  # sanity: models + dim

Behavior:

• Sources come from the git_sources registry (admin page — the source of truth): git rows are cloned (shallow, first time) or git pull --ff-onlyed; local rows are walked directly (a missing directory fails the run loudly); uploaded archives live under BOR_UPLOAD_DIR (zip-bomb guarded: absolute/../escaping-symlink members rejected; BOR_UPLOAD_MAX_MB caps compressed AND extracted bytes). BOR_GIT_SOURCES is the git-only fallback while the registry is empty. A failed source aborts the run — no partial junk.
• Delta: sha256 per (source, path) — unchanged files skipped (no re-embedding); changed files re-chunked + re-embedded with the two-phase upsert (one transaction per file — a failed embedding leaves the DB untouched).
• Prune: --prune removes docs of the imported sources whose files disappeared, no longer match the format filter, or now match a per-source ignore path.
• Summaries (phase 30): non-markdown files get a lite summary (best-effort — a failure still indexes the doc) + the position −1 summary chunk (at most one per document — replaced on re-import).
• After a KB-changing run: the lite-generated kb_overview is regenerated and sources_meta.version bumps exactly once (phase 53) — which is what marks older saved chats stale.
• Source removal: removing a registry row marks the source for cleanup on the next sync (its documents/chunks are pruned — phase 69).
• Formats per A9 (revised); hidden dirs/files skipped; exclusion list applied; per-source ignore paths (phase 89) applied as raw source-relative prefixes.

---

12. Roadmap (one story → one phase → one Playwright gate)

Phases live in .agents/phases/ — todo/ is empty; complete/ holds the phase directories (00_phase.md + task files) as read-only history. Next free phase number: 93.

Shipped (phases 01–92)

Range	Area	Highlights
01–10	v1 core	foundation; import story; chat RAG; honest deflection; suggestion chips; loading feedback; responsive polish; dark theme; hybrid retrieval; document viewer
11–16	answer + access	long answers; header consistency; viewer back-nav; localStorage persistence; steering notes; single-admin auth
17–28	UX + sources	thinking display; follow-bottom scroll (later revised, 42); shared header; mid-stream sources fix; thinking scroll fix; background animation; global tuning; git-based sources
29–36	sources + KB	tuning nav link; document summaries; KB overview prompt; admin sync button; cache busting; git-sources admin page; summary in viewer
37–41	agent tools	document tools ls/read; local directory sources; sync fail-fast model check; config/brand surfaces
42–49	chat behavior	no-reply autoscroll; thinking scroll-back; markdown tables; unlimited tool rounds; quadlet/j2 import; stop generation; archive uploads; retry answer
50–58	persistence + scale	saved chats; share links; payload limits; stale saved chats; cache-bust revalidation; save/share UX; any-extension import; summary editing; wide desktop column
59–66	docs push + polish	save-as-doc git push; sticky navbar; UI copy; ui customization env; unambiguous doc listing; sync/upload progress; bottom chat actions; history copy
67–75	reliability + tools	LLM retries; grep search tool; source removal cleanup; harness-aligned tools; scaffolding guardrails; teaching refusals; hidden-tab streaming; LLM chat history; full-session save
76–84	shell + security	SPA nav shell; navbar refresh; static background; API access tokens; history-driven suggestion chips; login rate limit; security headers; save payload limits; docs-push error sanitization
85–92	mobile + theming	mobile menu gate fix; history page width; big-read progress; mobile hamburger boot; per-source ignore paths; upload-without-scan; admin Theme tab; theme save + coverage

(Phase numbers 23/24/26/29–36/39/46/52 shipped in the pre-complete/ workflow era — their records live in .agents/reports/; every one of their features is in the running code and covered by a story E2E suite.)

Next up (from TODO.md — to be captured as phases via the

phase-authoring skill, one story/one phase/one E2E each)

1. Tree-style ls with folder summaries — ls behaves like a filesystem tree (top level = the synced projects, drill-down per folder) with per-folder summaries generated at sync time, so listing thousands of documents never floods the context. Run retrieval accuracy + performance checks against the current architecture.
2. Capped read for huge documents — a sensible per-read cap sized for the 128k-token minimum context of the owner's LLMs; the model is told the read was truncated and steered to grep for the rest; a visual indicator in the chat tells the user the read was truncated.

Completion (per phase — unchanged)

unit + integration green, coverage >90% on app/, the story's Playwright E2E green in isolation, UI verification against §7 passed, one --no-gpg-sign commit, phase dir moved to complete/.

---

13. Future (post-v1 hooks, deliberately not built)

• HNSW index on chunks.embedding at scale (sequential scan is fine today).
• Watchdog auto-re-import (inotify) — until then the script / Sync button is the truth.
• More LLM providers — the OpenAI-compatible client is the seam (BOR_LLM_BASE_URL).
• Multi-user accounts (tokens are the stop-gap; the require_user split is the seam).
• Vector-index-backed grep (trigram/GIN) if substring search gets slow on a large KB.