From 7baca3d289caab8c777d15a9ab83bba73b5d8024 Mon Sep 17 00:00:00 2001 From: ducoterra Date: Mon, 7 Sep 2026 14:26:26 -0400 Subject: [PATCH] update README --- README.md | 1013 +++++++++++++++++++---------------------------------- 1 file changed, 366 insertions(+), 647 deletions(-) diff --git a/README.md b/README.md index 1710451..0d4f576 100644 --- a/README.md +++ b/README.md @@ -1,6 +1,6 @@ # 🧠 Brain of Reese -A chippy, honest **RAG chatbot** over the `~/Homelab` and `~/Deployments` +A chippy, honest **RAG chatbot** over your `~/Homelab` and `~/Deployments` projects. Point it at your notes β€” markdown, YAML, JSON, Python, plain text β€” ask it anything, and it retrieves the relevant chunks with **hybrid search** (pgvector cosine βˆͺ Postgres full-text search, fused with @@ -8,222 +8,120 @@ Reciprocal Rank Fusion), feeds the **whole relevant document** to a **self-hosted LLM** (`turbo` via `https://aipi.reeseapps.com/v1`), and streams a grounded answer back. -If it doesn't have notes for your question, it admits it: -*"I haven't done anything like that"* β€” plus suggestions for what it **does** know. +If it doesn't have notes for your question, it admits it: *"I haven't done +anything like that"* β€” plus suggestions for what it **does** know. -> **Updated your notes?** The knowledge base is refreshed by re-running the -> import β€” it's idempotent and only re-embeds what changed: +> **Updated your notes?** Re-run the import β€” it's idempotent and only +> re-embeds what changed: > ```bash > uv run python -m scripts.import_docs --prune > ``` -> Details in [Updating the documents](#updating-the-documents). - -- **Stack:** FastAPI Β· Pydantic v2 Β· SQLAlchemy 2 Β· Alembic Β· pgvector Β· - vanilla HTML/CSS/JS (no CDN) Β· Playwright E2E -- **Planning:** architecture, LOCKED decisions and the phase roadmap live - in [`.agents/PLAN.md`](.agents/PLAN.md); per-story specs in - [`.agents/user_stories/`](.agents/user_stories/). +> See [Updating the documents](#updating-the-documents) for details. --- -## Development Setup +## Quick Start -### Prerequisites -- [uv](https://docs.astral.sh/uv/) -- [Podman](https://podman.io/) (with the `podman compose` provider) -- Node.js is **not** needed locally (asset minification happens in the - container build only) - -### 1. Install dependencies ```bash +# 1. Dependencies uv sync -``` -### 2. Configure -```bash +# 2. Configure (copy and edit) cp .env.example .env -# edit .env β€” the defaults already match the local compose setup. -# BOR_LLM_API_KEY: your aipi key (falls back to $AIPI_KEY if unset) -``` +# Set BOR_LLM_API_KEY β€” your aipi key (falls back to $AIPI_KEY) -### 3. Start the database (Postgres 17 + pgvector) -```bash +# 3. Start the database podman compose up -d db podman compose ps # wait until "healthy" -``` -### 4. Apply migrations -```bash +# 4. Apply migrations & import uv run alembic upgrade head -``` +uv run python -m scripts.import_docs -### 5. Import your knowledge base -```bash -uv run python -m scripts.llm_probe # sanity: models + 768-dim check -uv run python -m scripts.import_docs # import the configured sources (below) -``` - -Managed source kinds (one page, one registry) plus a manual override: - -- **Git sources** β€” managed on the **admin Git sources page** - (`/git-sources.html`) and stored in Postgres (see - [Git-based sources](#git-based-sources)). `import_docs` clones each repo - (first run) or pulls it (subsequent runs) into - `BOR_SOURCES_DIR//` (default `~/bor-sources`) and indexes the - checkouts. While the stored list is empty, the `BOR_GIT_SOURCES` variable - in `.env` is the fallback β€” the moment the page stores a source, the - variable is ignored. -- **Archive upload sources** (phase 49) β€” a `.tar`/`.tar.gz`/`.tgz`/`.zip` - uploaded on the *same* admin page (see - [Archive upload sources](#archive-upload-sources)). The archive is - unpacked under `BOR_UPLOAD_DIR//` and scanned immediately; it is - registered as a `kind=local` row, like a local directory. -- **Local directory sources** (phase 38) β€” an existing, non-git directory - on the server, registered on the *same* admin page (see - [Local directory sources](#local-directory-sources)). No clone, no - checkout copy: the directory is walked in place. There is **no env var - for local paths** β€” the DB is the registry. Since phase 49 the page's - β€œAdd a local directory” form is gone (the archive upload replaced it): - a plain directory is registered via `POST /api/git-sources` with - `kind=local`; existing Local rows are unchanged. -- **Manual directories** β€” `--source ` (repeatable) imports local - directories directly and *always wins* over the stored sources (git and - local) and the env fallback. -- If neither is set (stored list, `--source`, and `BOR_GIT_SOURCES` all - empty), `import_docs` falls back to the **previous** default, - `~/Homelab` + `~/Deployments` β€” kept only for backwards compatibility, - now replaced by the managed sources; the UI Sync button instead fails - loudly ("no sources configured (git or local)"). - -### 6. Run the app -```bash +# 5. Run the app uv run uvicorn app.main:app --reload -# β†’ http://localhost:8000 (chat) http://localhost:8000/sources.html (KB) +# β†’ http://localhost:8000 (chat) ``` -> πŸ“ **After this, day-to-day is just: edit markdown β†’ re-run the import.** -> See [Updating the documents](#updating-the-documents) below. +**Stack:** FastAPI Β· Pydantic v2 Β· SQLAlchemy 2 Β· Alembic Β· pgvector Β· +vanilla HTML/CSS/JS (no CDN) Β· Playwright E2E -## Using the UI +--- -- **Chat** (`/`) β€” ask questions; answers stream in with **source chips** - that cite the exact documents used. Clicking a chip opens that document - in an **almost-fullscreen modal** on the same page (no new tab). The - onboarding suggestion chips follow the last 3 questions asked β€” on a - fresh deployment (no saved questions yet) they seed from - `BOR_SUGGESTIONS`. -- **Document viewer** β€” the modal above *is* the viewer; the full text of - any indexed document is served from the database (no filesystem access): - markdown is rendered, every other format (`yaml`, `json`, `py`, `txt`, …) - is shown as escaped monospace text. `/document.html?source=…&path=…` - stays as the **full-page / direct-link** form (the modal's β€œFull page” - button and the URL to share β€” it works without JS). Unknown documents - get a designed not-found state with a link back to the index. -- **Sources** (`/sources.html`) β€” the indexed document list; the *Path* - column opens each document in the same **almost-fullscreen modal** (no - new tab). **Admin-only** β€” anonymous visitors see a sign-in gate instead - (the catalog is what the admin login locks; the document viewer asks - for an access token β€” see [API tokens](#api-tokens). Shared chats stay - open to everyone). -- **Git sources** (`/git-sources.html`) β€” the admin-managed source - registry: the git repositories the **Sync sources** button clones and - indexes, uploaded archives it unpacks and scans, **and** existing local - directories it imports directly (one table with a `kind` discriminator, - one page); **admin-only** (the same sign-in gate as Sources). Add or - remove sources here β€” no `.env` editing, no restart. The **archive - upload form** (phase 49) accepts `.tar`, `.tar.gz`, `.tgz`, `.zip`: the - archive is unpacked under `BOR_UPLOAD_DIR//` (name = filename - minus the archive suffix) and scanned immediately β€” re-uploading the - same filename replaces that source **in place** (one folder, one row, - dropped files pruned; see - [Archive upload sources](#archive-upload-sources)). A local directory - must be an absolute, existing directory at add-time (a - missing/relative path is rejected, naming the path; so are duplicates) - β€” since phase 49 this is an API-only operation (`POST /api/git-sources` - with `kind=local`; the page's form was replaced by the upload form). List - rows carry a **Git** or **Local** badge. Adding does not clone: the - Sync button performs that (git + local together, one run, prune over - the union) and still prunes **upstream file churn** β€” a file deleted - in a repo or dropped from a local directory leaves the index on that - run. **Removing a source is a total removal, done immediately** β€” a - confirmation modal states it first, then the source's entry, all of - its indexed documents, and β€” for git clones and uploaded archives β€” - its files on disk (the checkout under `BOR_SOURCES_DIR` or the - unpacked folder under `BOR_UPLOAD_DIR`) are gone in one action; files - in the owner's own local directories are never touched. +## Pages at a glance -## Thinking +The app is a single-page application β€” all views load from `/` with a +client-side router. Direct URLs deep-link to the matching view. -The self-hosted `turbo` model reasons before it answers. That reasoning is -streamed with the turn as `thinking` SSE events and shown in a -**collapsible "Thinking" block** above the answer bubble: it opens and -fills in live while the model thinks, tucks itself away the moment the -first answer token lands, and stays click-toggleable afterwards. Thinking -persists with the message, so a reloaded conversation restores the block -(collapsed) alongside the answer. How much the model thinks β€” or whether -it thinks at all β€” is the model's call: turns without reasoning render -exactly as before. +| Page | URL | Who can see it | +|------|-----|----------------| +| **Chat** | `/` | Everyone (token gate for non-shared) | +| **RAG / Knowledge base** | `/sources.html` | Admin only | +| **Git sources** | `/git-sources.html` | Admin only | +| **Tuning** | `/tuning.html` | Admin only | +| **Saved chats** | `/history.html` | Admin only | +| **Access tokens** | `/tokens.html` | Admin only | +| **Document viewer** | `/document.html?source=X&path=Y` | Token users & admins | +| **Login** | `/login.html` | Everyone | +| **Shared chat** | `/shared/` | Everyone (anonymous) | +| **Edit doc** | `/doc-edit.html` | Admin only | -To hide it, set `BOR_STREAM_THINKING=0` β€” the `thinking` events stop -(the per-turn log line still counts `thinking_chars`). +### Chat (`/`) -## Agent document tools (ls + read + grep) +Ask questions; answers stream in with **source chips** that cite the exact +documents used. Clicking a chip opens the document in an **almost-fullscreen +modal** on the same page (no new tab). The onboarding suggestion chips +follow the last 3 questions asked β€” on a fresh deployment they seed from +`BOR_SUGGESTIONS`. -Retrieval only puts the top documents in context. When an answer depends -on a file a note *references* ("the exact JSON shape is in -example-record-file.json"), the model can extend its own context with -three server-side tools β€” on **grounded** (high-relevance) turns only. -The surface mirrors the shape the chat model was trained on (the pi.dev -harness tools, owner decision 2026-09-03), and the canonical document -identity everywhere is the combined `source/path` string: +### RAG / Knowledge base (`/sources.html`) -* **`ls`** β€” lists every indexed document, one `source: X | path: Y | - title: Z` line each (the same order as the Sources page); pass a - source name as `path` to list one source's documents; -* **`read(path)`** β€” appends the **full** text of one more indexed - document to the context (never truncated); `path` is the combined - `source/path` string exactly as shown in the `ls` output; -* **`grep(pattern, path?)`** β€” searches the indexed documents for an - exact string (case-insensitive fixed substring) and returns up to 20 - matching `source/path:line: text` lines; an optional `path` limits the - search to one document. A locator, not a context-adder: it never adds - to the answer context β€” the model `read`s the winner. +The indexed document list. The *Path* column opens each document in the same +**almost-fullscreen modal**. Admin-only β€” anonymous visitors see a sign-in +gate instead. -Each call the model requests is executed against Postgres only (no extra -LLM round trip) and streamed as an SSE `tool` frame ahead of the answer β€” -`{"type": "tool", "name": "ls" | "read" | "grep", "argument": … | null}` -(`argument` is the single string the model passed β€” `read`'s `path`, -`grep`'s `pattern`, `ls`'s scope β€” or null). In the chat, each call shows -a transient **calling-tool status** in addition to "thinking" (the send -button keeps its busy state β€” "Stop" β€” for the whole turn) and a visible -tool line (`πŸ”Ž Listing documents` / `πŸ“„ Reading source/path` / -`πŸ”Ž Searching for pattern`) lands above the answer, one per call, in -order. The tool lines persist with the message, so a reloaded -conversation re-renders them. The read document is reflected in the -answer's **source chips** and in the `query_log` row. +### Git sources (`/git-sources.html`) -The tools stay offered for the whole turn β€” the model may call them as -many times as it needs (re-lists included), bounded only by a round cap -that stops a pathological infinite loop: +The admin-managed source registry: add or remove git repositories, upload +source archives (`.tar`, `.tar.gz`, `.tgz`, `.zip`), and register local +directories β€” all in one table with a `kind` discriminator. No `.env` +editing, no restart. The **Sync sources** button clones/pulls/walks every +source and re-imports in one click. -| Env | Default | Meaning | -|---|---|---| -| `BOR_AGENT_MAX_ROUNDS` | `10` | hard cap on agent tool rounds per grounded turn β€” every call the model emits consumes a round; at the cap the loop forces one final no-tools answer | +### Tuning (`/tuning.html`) -`BOR_AGENT_MAX_ROUNDS=0` reproduces the pre-agent chat behavior exactly -(no `tools` in the request, no `tool` frames) β€” the kill switch. -Deflected turns run no tools at all β€” the low-relevance path is -unchanged. +Set global instructions that are injected into the system prompt of every +chat turn. Add notes like *"be more concise"* or *"assume I'm on NixOS"*. +List, edit, or remove them here β€” no conversation required. -## Admin & sign-in +### Saved chats (`/history.html`) + +Every conversation is saved automatically. Click a title to return to that +chat. Admin-only view. + +### Access tokens (`/tokens.html`) + +Generate tokens and hand them out β€” a token opens chat, the answers, and +the documents they cite. Shared chats stay open to everyone. + +### Shared chat (`/shared/`) + +A read-only snapshot of any conversation, shareable via a link. Open to +everyone β€” no login or token required. + +### Edit doc (`/doc-edit.html`) + +Review and adjust an AI-generated answer before committing it to the docs +repository. Admin-only flow page. + +--- + +## Admin & Sign-in Brain of Reese has exactly **one account: the admin (you)**. Signing in unlocks the **full Sources catalog**, the **answer-tuning** controls, and -**token management**. **Shared chats** are the only content that stays -open to anonymous visitors; everyone in between gets an **API token** β€” -the admin generates one in the **Tokens** view and hands it out, and the -holder enters it at the in-app gate to use chat and open the documents -answers cite (see [API tokens](#api-tokens) below). +**token management**. **Shared chats** are the only content that stays open +to anonymous visitors. ### Setup (one-time) @@ -237,418 +135,199 @@ BOR_SESSION_SECRET= # signs the session cookie ``` **Fail-loud:** while either variable is empty the app refuses to start, -naming the missing one(s): +naming the missing one(s). -``` -RuntimeError: Brain of Reese cannot start: admin auth is not configured. -Set the missing variable(s): BOR_ADMIN_PASSWORD, BOR_SESSION_SECRET … -``` +### How authentication works -### How it works +| Endpoint | Purpose | +|----------|---------| +| `POST /api/login` | Password β†’ signed `bor_session` cookie | +| `POST /api/logout` | Clears session cookie | +| `GET /api/whoami` | `{"authenticated": bool, "role": "admin"|"user"|"anonymous"}` | +| `POST /api/token-auth` | Token (`bor_…`) β†’ same cookie, role `user` | -- `POST /api/login {"password": …}` β†’ `204` + signed `bor_session` cookie - (Starlette `SessionMiddleware` β€” an itsdangerous-signed cookie, no - server-side store, no new service, no DB table); any mismatch β†’ `401` - `{"detail": "invalid password"}` (constant-time compare, one generic - message β€” no user enumeration, there is only one user). -- `POST /api/logout` β†’ `204` (session cleared and cookie expired; - idempotent for anonymous callers). -- `GET /api/whoami` β†’ `{"authenticated": bool, "role": "admin"|"user"|"anonymous"}` - β€” the single source of truth for every UI gating decision - (`authenticated` is true for **both** admin and token users; the UI - gates on `role === "admin"`). -- `POST /api/token-auth {"token": "bor_…"}` β†’ `204` + the **same signed - cookie** (role `user`); malformed / unknown / revoked all get one - generic `401` `{"detail": "invalid token"}` β€” no enumeration. The two - roles coexist in one session; **Sign out** clears both at once. -- Cookie flags: `same_site="lax"`, `https_only` off β€” **no HTTPS - enforcement on purpose** (homelab HTTP; the cookie is single-admin - convenience, not a cloud boundary). Max age `BOR_SESSION_MAX_AGE` - (default `43200` = 12 h, refreshed while active). -- Sign in from the chat header (**Sign in**) or `/login.html` directly; - the header then offers **Sign out** (logout + reload). +The cookie uses `same_site="lax"`, `https_only` off β€” **no HTTPS +enforcement** (homelab HTTP; the cookie is single-admin convenience, not a +cloud boundary). Max age is `BOR_SESSION_MAX_AGE` (default `43200` = 12 h). -### Who can do what - -| Capability | Anonymous | Token user | Admin (signed in) | -|---|---|---|---| -| Shared chats (`/shared/`) | yes | yes | yes | -| Chat (`/`) + suggestion chips | token gate | yes | yes | -| Document viewer (`/document.html?source=…&path=…`) | token gate | yes | yes | -| Sources catalog (`/sources.html`, `GET /api/docs`) | sign-in gate | 403 | full catalog | -| Tuning (Tune button, Tuning panel, `/api/steering`) | UI hidden | 403 | full | -| Saved-chat history (`/history.html`) | sign-in gate | no History view | full | - -### API tokens +### Access tokens The admin can hand out access without sharing the admin password. -- **Generate:** sign in as admin β†’ **Tokens** in the navbar - (`/tokens.html`). Type a label (e.g. `alice`) and hit **Generate** β€” a - `bor_` + 32-hex token appears in the *shown once* block. Copy it now: - only its SHA-256 hash is stored, so the plaintext can never be - retrieved again. -- **Use:** the holder opens the app (or a direct document URL) and gets - the **token gate** instead of content β€” they enter the token, and the - browser caches it in `localStorage["bor.token"]`, silently re-sending - it on every page load (no re-entry on reloads or new tabs). **Sign - out** clears the cache; a failed silent re-auth (e.g. a revoked token) - drops the cached copy and shows the gate again. Private-mode browsers - still work β€” the gate just can't cache. -- **Scope:** a token user can chat (with suggestion chips) and open the - documents answers cite β€” nothing else. The Sources catalog, git - sources, tuning, document drafts, and the saved-chat history stay - admin-only (`403` on their APIs; no History view). **Shared chats - stay open to everyone** β€” the only anonymous content. -- **Revoke:** **Revoke** on the token's row (inline two-step confirm). - Revocation is **immediate**: the token's next request β€” including a - fresh login attempt β€” is refused with the same generic `401` as a bad - token. Revoked tokens stay in the list, marked **Revoked**, with their - last used time. +1. **Generate:** sign in β†’ **Tokens** in the navbar. Type a label and hit + **Generate** β€” a `bor_` + 32-hex token appears in the *shown once* block. + Copy it now: only its SHA-256 hash is stored. +2. **Use:** the holder enters the token at the in-app gate. The browser + caches it in `localStorage["bor.token"]` for silent re-auth on reloads. +3. **Revoke:** click **Revoke** on the token's row. Revocation is immediate. -The public API endpoints stay stateless β€” the signed cookie is the only -session state in the system. +A token user can chat and open documents β€” nothing else. The Sources +catalog, git sources, tuning, and history stay admin-only. -## Tuning your answers +--- -*Admin-only* β€” sign in first (see **Admin & sign-in** above); anonymous -visitors never see the Tune button or the Tuning panel. +## Using the App -If an answer isn't quite right β€” too chatty, wrong assumption, missing -context β€” **tune** Brain right there: +### Updating the documents -1. Press **β€œTune”** in the meta row under any completed answer (deflected - ones included). -2. Type a short instruction (1–2000 chars), e.g. *β€œbe more concise”* or - *β€œassume I'm on NixOS”*, and **Save**. - -The note is stored in Postgres (`steering_notes`) and read into the -**system prompt of every subsequent chat turn** as a `` section -(numbered, oldest first, capped at `BOR_STEERING_MAX_CHARS` chars β€” -default 8000, overflow marked `[…truncated…]`). With no stored notes the -prompt is byte-identical to the un-tuned one, so tuning is opt-in per -note. - -List or remove notes at any time from the **β€œTuning”** button in the chat -header (count badge, newest-first, per-note delete). The API is stateless -JSON if you prefer curl: +The knowledge base is refreshed by **re-running the import**. It is +idempotent and delta-based (sha256 per file), so a refresh after a normal +editing session takes seconds: ```bash -curl -s localhost:8000/api/steering # list (newest first) -curl -s -X POST localhost:8000/api/steering \ - -H 'Content-Type: application/json' -d '{"note": "be more concise"}' -curl -s -X DELETE localhost:8000/api/steering/ # remove -``` - -## Updating the documents - -**This is the workflow you'll use most.** The knowledge base is refreshed by -**re-running the import**. It is idempotent and delta-based (sha256 per -file), so a refresh after a normal editing session takes seconds: - -```bash -# After editing/adding/removing notes in your projects: +# After editing/adding/removing notes: uv run python -m scripts.import_docs # re-index what changed -uv run python -m scripts.import_docs --prune # also drop deleted/out-of-scope files +uv run python -m scripts.import_docs --prune # also drop deleted files # Point it at extra directories (repeatable): uv run python -m scripts.import_docs --source ~/SomeOtherDocs ``` -With **git-based sources** (below) each run first pulls the latest commits -of your repos, so this same command is the whole update loop: commit in the -repo β†’ re-run the import. +With **git-based sources** each run first pulls the latest commits of your +repos, so this same command is the whole update loop: *commit β†’ re-run*. Then check the **Sources** page (`http://localhost:8000/sources.html`): the *documents* / *chunks* counters and *last indexed* timestamp should -reflect the new files, and each document row shows when it was last -embedded. +reflect the new files. -- The import prints one line per file (`import: added|updated|unchanged| - pruned …`) and ends with a greppable summary (`import: summary files=… - added=… updated=… unchanged=… pruned=… chunks=… embed_batches=… - formats=md:203,yaml:267,…`), so it is safe to run from a cron job or - after every commit. -- Indexed formats (A9, revised 2026-08-27): **`md, markdown, txt, - yaml, yml, json, py`**, the Podman quadlet family (**`container, - network, volume, image, pod, kube, swap, os, endpoint`**), and **`j2`** - Jinja templates (case-insensitive; narrow with `BOR_IMPORT_EXTENSIONS`). - Any path with a - **dot-prefixed component** β€” hidden files or vendored caches like - `.esphome/.espressif/**` β€” is skipped, along with `.venv`, - `node_modules`, `.git`, `__pycache__`, `.pytest_cache`, `dist`, `build`. - `--prune` also drops documents whose files no longer match the filter β€” - that's how previously imported junk leaves the index. -- Non-markdown files get format-aware chunking (YAML top-level keys / - `---` docs, JSON top-level keys, Python top-level defs/classes via - stdlib `ast`; quadlet unit files and `j2` templates are paragraph- - packed as plain text) and their title comes from the file stem. -- Unchanged files are **not re-embedded** β€” only new/changed ones, so - refreshes are cheap. -- After a run that **changed** the knowledge base (at least one document - added or updated), the import also regenerates the stored **KB - overview** β€” a plain-text outline of the KB's basic categories that - every chat turn injects into the system prompt as `` - (phase 31). The regeneration is best-effort and change-gated: unchanged - re-imports and `--limit` debug runs skip it (no `lite` call), and a - `lite`-model failure leaves the previous outline intact without failing - the import. The summary line ends `overview=updated|skipped|failed`. -- To sanity-check the LLM backend (models + embedding dimension) after any - aipi change: `uv run python -m scripts.llm_probe`. +- The import prints one line per file and ends with a greppable summary, + so it is safe to run from cron or after every commit. +- Indexed formats: **md, markdown, txt, yaml, yml, json, py**, the Podman + quadlet family (**container, network, volume, image, pod, kube, swap, + os, endpoint**), and **j2** Jinja templates (case-insensitive). +- Hidden files and common cache dirs (`.venv`, `node_modules`, `.git`, + `__pycache__`, etc.) are skipped. +- Non-markdown files get format-aware chunking (YAML keys, JSON keys, + Python classes/defs). ### Git-based sources -Rather than pointing the import at local folders, point it at **git -repositories** β€” the notes live in the repos and `import_docs` keeps local -checkouts of them up to date for you. - -**Where the list lives (phase 35):** the primary management surface is -the **admin Git sources page** (`/git-sources.html`) β€” add or remove -repositories there and the list is stored in Postgres (the `git_sources` -table). `BOR_GIT_SOURCES` in `.env` is the **empty-table fallback**: it -only applies while the stored list is empty, and is ignored once the page -has any row (the page becomes the source of truth β€” no `.env` editing, no -restart needed afterwards). +Rather than pointing the import at local folders, point it at +**git repositories**. The admin **Git sources page** (`/git-sources.html`) +is the primary management surface β€” add or remove repositories there and +the list is stored in Postgres. ```env # .env β€” the fallback list (fresh setups, or until the admin page -# stores a source; phase 35 demotes this variable, it does not remove it) +# stores a source; the page becomes the source of truth) BOR_GIT_SOURCES=https://git.reeseapps.com/reese/homelab.git,git@github.com:reese/deployments.git BOR_SOURCES_DIR=~/bor-sources # default; each repo lands in // ``` -- The effective list (stored rows, else `BOR_GIT_SOURCES` while the stored - list is empty) is a set of git repo URLs. Auth is whatever the machine - supplies β€” `https://…` via the OS credential helper, or `git@host:repo.git` - via your SSH key; no credentials are stored in the app or `.env`. - Stored URLs are shape-validated on the page (`https://`, `ssh://`, - `git@…` β€” scp-style `host:repo` is rejected). -- Every run **clones** each repo (first time, shallow `--depth 1`) or - **pulls** it (`git pull --ff-only` β€” fast-forward only, so a diverged or - broken checkout fails loudly instead of merging) into - `BOR_SOURCES_DIR//`, then indexes the checkouts exactly like - any local directory (A9 format filter, hidden-dir skip, sha256 delta). - `documents.source` is the repo directory name (e.g. `homelab`). -- **`--source ` overrides**: when the flag is given, the git - sources (stored list *and* `BOR_GIT_SOURCES`) are ignored and the - manual directory(ies) are imported. -- **A failed sync aborts the run**: if any repo cannot be cloned/pulled, - `import_docs` exits non-zero naming the failing repo and imports - **nothing** (no partial junk). Fix the URL/connectivity and re-run β€” the - other checkouts stay on disk and are pulled as usual. - -### Local directory sources - -Not every set of notes lives in a git repo β€” a plain directory can be a -first-class source too (phase 38). It shares the git sources' **one -table** (the `git_sources` registry with a `kind` discriminator: `git` | -`local`, migration 0007), **one admin page**, and **one Sync button**: - -- **Register it via the API (phase 49)** β€” the phase-38 β€œAdd a local - directory” form on the Git sources page was replaced by the archive - upload form; adding a plain directory is now an **API-only** operation: - `POST /api/git-sources` with `{"kind": "local", "path": …}`. Add-time - validation fails loud: the path is trimmed, `~` is expanded, and must be - an **absolute, existing directory on the server** β€” anything else - (missing, relative, a file) is 422 with the path named; a duplicate path - is 409 the same way. **Existing Local rows are unchanged**: they still - list, remove, and sync exactly as before. There is **no env var for - local paths** β€” the DB is the local-source registry - (`BOR_GIT_SOURCES` stays a git-only fallback). -- **Sync walks it directly** β€” no clone, no checkout copy: each run - indexes the directory in place (A9 format filter, hidden-dir skip, - sha256 delta), together with the git checkouts in the **same run**. - `documents.source` is the directory's name. The directory is - re-verified to exist **at sync time** (it may have moved or been - deleted since add-time): a missing directory fails the run loudly, - naming the path, and imports **nothing** (the same pre-import fail-loud - as a failing git clone). -- **Pruning is over the union** β€” git checkouts and local directories are - imported together with `prune=True`, so a file removed from a local - directory or a repo leaves the index on that run (upstream file - churn). Removing the source on the page is a **total removal, done - immediately** (a confirmation modal states it first): its entry and - its indexed documents leave at once β€” but the directory itself is the - owner's own and is **never touched on disk** (only git checkouts and - uploaded archives get their files deleted). -- **`import_docs`** (no `--source`) resolves the stored git **and** local - rows β€” git cloned/pulled as above, local walked directly β€” in one run; - `--source` still wins over everything; while the table is empty, - `BOR_GIT_SOURCES` is the git-only fallback; no git rows, no local rows, - and no env URLs fails loudly ("no sources configured (git or local)"). +- Auth is whatever the machine supplies β€” HTTPS via the OS credential + helper, or SSH via your key; no credentials are stored in the app. +- Every run **clones** (first time, shallow `--depth 1`) or **pulls** + (`git pull --ff-only`) each repo, then indexes the checkout. +- A failed sync aborts the run β€” no partial junk. ### Archive upload sources -Upload a `.tar`, `.tar.gz`, `.tgz`, or `.zip` archive to make it a -source (phase 49, owner permission 2026-08-28). The form on the admin -Git sources page β€” and the `POST /api/git-sources/upload` route behind -it β€” replaced the phase-38 β€œAdd a local directory” form. An uploaded -source is registered as a `kind=local` row, so everything local -directory sources do (Sync, prune, remove) applies to it: +Upload a `.tar`, `.tar.gz`, `.tgz`, or `.zip` archive to make it a source. +The form on the Git sources page β€” and the `POST /api/git-sources/upload` +route behind it β€” accepts archives and unpacks + scans them immediately. -- **Accepted formats:** `.tar`, `.tar.gz`, `.tgz`, `.zip` β€” anything else - is 422 naming the accepted set. Unpacking is guarded: absolute member - paths, `..` traversal, symlink/hardlink targets escaping the unpack - folder, and device/FIFO members are rejected (422), and the total - *extracted* bytes count against the size cap (zip-bomb guard). A - zero-entry archive is 422; an archive with only non-A9 files is a - **valid replacement** (it indexes nothing and prunes the source's - previous documents). -- **Naming rule:** the source name is the **filename minus the archive - suffix** (`homelab.tar.gz` β†’ `homelab`, case-sensitive). The name is - both the folder under `BOR_UPLOAD_DIR` and the row's identity; files - land in the KB exactly as packed (no auto-unwrap of a single top-level - folder). -- **In-place replace:** re-uploading the same filename creates **no - second folder and no second row** β€” the new content is unpacked to a - temp sibling and atomically renamed over the existing folder (no - missing window; a failed upload never touches the existing folder, - row, or KB), the row is upserted by path (`kind='local'`, - `added_at` preserved), and the source is re-scanned with `prune=True` - β€” files dropped from the archive leave the index in the same request. -- **The scan is synchronous in the request:** it fails fast on the - models (503 when they are down β€” the folder/row are already committed, - so the next sync or re-upload retries idempotently), then runs the - single-source import (embeddings + per-document summaries) and the - change-gated KB overview refresh, and answers 200 with the sync-style - counts (`added`, `updated`, `unchanged`, `pruned`, …) the page renders - as its result line. One upload at a time β€” a concurrent upload gets - 409. -- **Where + how big:** archives unpack under `BOR_UPLOAD_DIR` (default - `~/bor-sources/uploads` β€” deliberately separate from the git checkouts - in `BOR_SOURCES_DIR`); `BOR_UPLOAD_MAX_MB` (default 512) caps **both** - the compressed upload and the total extracted bytes. +- Re-uploading the same filename replaces the source **in place** (no + second folder, no duplicate row). +- Zip-bomb guard: absolute member paths, `..` traversal, and symlinks + escaping the unpack folder are rejected. +- Archives unpack under `BOR_UPLOAD_DIR` (default `~/bor-sources/uploads`). +- `BOR_UPLOAD_MAX_MB` (default 512) caps both compressed and extracted size. + +### Local directory sources + +A plain directory can be a first-class source too. It shares the git +sources' **one table** (the `git_sources` registry with a `kind` +discriminator: `git` | `local`), **one admin page**, and **one Sync button**. + +- Register via API: `POST /api/git-sources` with `{"kind": "local", "path": …}`. +- Sync walks it directly β€” no clone, no checkout copy. +- The directory is re-verified to exist at sync time; a missing directory + fails the run loudly. ### Sync from the UI -The **Sync sources** button on the **Sources** page β€” visible to the -**admin only** (anonymous visitors never see it) β€” runs the whole -git-source refresh in one click, in-process: +The **Sync sources** button on the **Sources** page (admin only) runs the +whole git-source refresh in one click: -1. **clone/pull + walk** every configured source β€” the git sources (the - admin-managed `git_sources` table; `BOR_GIT_SOURCES` only while that - list is empty) through the same `clone_or_pull` the CLI uses (shallow - clone on first run, `git pull --ff-only` afterwards), **and** the - local directories registered on the same page β€” including uploaded - archives (their `BOR_UPLOAD_DIR//` folders are `kind=local` - rows, *Archive upload sources*) β€” walked directly (re-verified to - exist at sync time β€” a missing directory fails the run loudly, naming - the path); -2. **re-import with prune** β€” the `--prune` equivalent, so files deleted - upstream leave the index (the button is the canonical "mirror the - repos" action); the sha256 delta still skips unchanged files, so an - unchanged re-sync re-embeds nothing; +1. **clone/pull + walk** every configured source β€” git repos and local + directories, including uploaded archives. +2. **re-import with prune** β€” files deleted upstream leave the index. 3. **regenerate the KB overview** (the `` outline every - chat turn injects) β€” but only when the import actually changed the - knowledge base. + chat turn injects) β€” only when the import changed the knowledge base. -- **Prerequisites:** at least one source must be configured β€” a git or - local row on the admin Git sources page, or `BOR_GIT_SOURCES` in `.env` - while the stored list is empty (git-only); **all** empty fails the sync - loudly ("no sources configured (git or local)"), because the button - targets the admin-managed registry (manual `--source` directories have - no place in it) β€” and `git` must be on the app's `PATH` for git sources. -- **States:** clicking starts the run (`202`) and the button goes - disabled with **Syncing…** (spinning icon) while the page polls - `GET /api/sync/status` every 2 s. There is deliberately **no - client-side timeout** β€” a clone + embed can legitimately take minutes, - so the poll is the feedback loop and the server state is authoritative. - On success the button settles to **Synced HH:MM** with the last result - in a live region (`1 added`, `0 added Β· 1 unchanged`, …); on failure it - re-enables (retry-ready) and a red error banner names the failure (git's - stderr, with any embedded credentials masked). -- **One sync at a time:** a second trigger while a run is in flight gets - `409` ("a sync is already running"); the UI adopts the in-flight run - instead of starting a second one, and a page reload mid-sync re-attaches - to it the same way. -- **Idempotent:** re-syncing unchanged repos is a no-op β€” fast-forward - pull, hash skip, and the overview is left alone (its regeneration is - change-gated). +- States: the button shows **Syncing…** with a spinner while polling + `GET /api/sync/status` every 2 s. No client-side timeout β€” a clone + + embed can legitimately take minutes. +- One sync at a time: a second trigger while a run is in flight gets a + `409`. -## Caching / deploys +--- -A deploy is a commit β€” and the browser must see it **without a hard -refresh** (the "the pages are too sticky" problem, phase 33). One -Starlette middleware (`app/core/caching.py`) applies the rule at the -transport layer: +## Thinking -- **HTML pages always revalidate β€” and never 304.** Every page - (`/`, `/index.html`, `/sources.html`, `/document.html`, `/login.html`, - `/tuning.html`, `/git-sources.html`, `/history.html`, `/shared.html`, - plus the dynamic share page `/shared/`) ships - `Cache-Control: no-cache`, **no `etag`, no `last-modified`**, so each - visit re-checks the page with the server and always gets a fresh 200 - body β€” a page never lingers in the browser's cache unchecked, and a - revalidation can never be answered "not modified" (see *Cache busting - below* for why). -- **Assets are versioned and cached for a year.** The pages reference - their CSS/JS with a token (`/assets/styles.css?v=`), and every - `/assets/*` response ships - `Cache-Control: public, max-age=31536000, immutable`. The token is what - identifies the content, so long-term caching is safe: a new token means - a new URL, which the browser fetches fresh. A conditional `GET` on a - versioned asset URL may still be answered `304` β€” the URL already - encodes the version, so that is safe. -- **The token is the deploy.** In a git checkout (the normal case) it is - the short SHA of `HEAD` (`git rev-parse --short HEAD`), computed once - per process start β€” so **every commit/deploy flips the token** and the - versioned asset URLs change with it. A checkout without `.git` (or a git - failure) falls back to a stable content hash of the `frontend/` tree - (sorted path + mtime + size), so dev checkouts still bust; a missing - static dir gets the placeholder token `dev`. -- **The API is untouched.** Nothing under `/api/*` β€” the SSE chat stream - in particular β€” gains or loses a header or has its body read; the SSE - endpoint's own `Cache-Control: no-cache` is set by the endpoint itself. +The self-hosted `turbo` model reasons before it answers. That reasoning is +streamed with the turn as `thinking` SSE events and shown in a +**collapsible "Thinking" block** above the answer bubble: it opens and +fills in live while the model thinks, tucks itself away the moment the +first answer token lands, and stays click-toggleable afterwards. -No CDN, no new services, no build-step change: the middleware rewrites -the asset references of the known pages in flight. The unversioned -asset paths keep working too (the static mount ignores the query string), -so old tabs and direct links to `/assets/…` still resolve. +To hide it, set `BOR_STREAM_THINKING=0`. -> **Deploy note:** the very first deploy onto this scheme needs one -> normal page visit, so the browser revalidates the HTML once and starts -> requesting the versioned assets; every commit after that is picked up -> automatically. +--- -### Cache busting: why pages never 304 (phase 54) +## Agent document tools (ls + read + grep) -The `?v=` token flips on the **next process start** β€” a -commit is a deploy, so the restarted server's HTML references new asset -URLs, and the browser fetches them fresh into its year-long asset cache. +Retrieval only puts the top documents in context. When an answer depends on +a file a note *references* ("the exact JSON shape is in +`example-record-file.json`"), the model can extend its own context with +three server-side tools β€” on **grounded** (high-relevance) turns only: -- **`/assets/*`** is cached `immutable` for a year *under the versioned - URL* β€” a conditional `GET` may 304, because the URL already encodes - the version. -- **HTML pages** are served `no-cache` and **never 304**, publishing no - `etag` / `last-modified`. The reason: the page body the browser - receives is *rewritten per process* (its asset refs gain - `?v=`), so the static file's upstream validators would describe - the *file*, not the *bytes served* β€” a conditional `GET` matching them - would 304 out of the rewrite and leave the browser on HTML pointing at - the **previous** commit's CSS/JS, which the year-long asset cache - serves until a hard reload (the hole measured in phase 54). Pages - therefore revalidate against the bytes actually served: always a full - 200. +| Tool | Description | +|------|-------------| +| **`ls`** | Lists every indexed document (`source: X \| path: Y \| title: Z`); pass a source name as `path` to list one source | +| **`read(path)`** | Appends the **full** text of one indexed document to context (never truncated); `path` is the combined `source/path` string | +| **`grep(pattern, path?)`** | Searches indexed documents for an exact string (case-insensitive fixed substring); returns up to 20 `source/path:line: text` matches; an optional `path` limits to one document | -**Local development:** a `git` commit changes the token on the next -server restart. If a browser still shows an old layout after a restart, -hard-reload once (`Ctrl/Cmd-Shift-R`) β€” the phase-54 fix guarantees the -*next* navigation is current, but it cannot un-pin a document that a -pre-54 deploy already 304'd into the browser's cache. +Each call is executed against Postgres only (no extra LLM round trip) and +streamed as an SSE `tool` frame. In the chat, each call shows a transient +**calling-tool status** alongside "thinking" (the send button keeps its +busy state β€” "Stop" β€” for the whole turn). -## Checking retrieval quality +The model may call tools as many times as needed, bounded by a round cap: -Ask the *real* pipeline (live aipi embeddings + the current KB) whether a -question lands on the right document, with the gate verdict and per-document -cosine / FTS / fused scores: +| Env | Default | Meaning | +|-----|---------|---------| +| `BOR_AGENT_MAX_ROUNDS` | `10` | hard cap on agent tool rounds per grounded turn (0 = no tools, the kill switch) | + +--- + +## Tuning your answers + +*Admin-only* β€” sign in first. If an answer isn't quite right β€” too chatty, +wrong assumption, missing context β€” **tune** Brain right there: + +1. Press **"Tune"** in the meta row under any completed answer (deflected + ones included). +2. Type a short instruction (1–2000 chars), e.g. *"be more concise"* or + *"assume I'm on NixOS"*, and **Save**. + +The note is stored in Postgres (`steering_notes`) and read into the +**system prompt of every subsequent chat turn** as a `` section +(numbered, oldest first, capped at `BOR_STEERING_MAX_CHARS` chars β€” default +8000, overflow marked `[…truncated…]`). + +List or remove notes from the **"Tuning"** button in the chat header (count +badge, newest-first, per-note delete). The API is stateless JSON if you +prefer curl: ```bash -uv run python -m scripts.eval_retrieval "How did I install gitlab?" -uv run python -m scripts.eval_retrieval --from-file questions.txt --top 8 +curl -s localhost:8000/api/steering # list +curl -s -X POST localhost:8000/api/steering \ + -H 'Content-Type: application/json' -d '{"note": "be more concise"}' +curl -s -X DELETE localhost:8000/api/steering/ # remove ``` -Requires `AIPI_KEY` in the environment (same convention as -`scripts/llm_probe.py`) and an imported knowledge base. +--- ## How retrieval works (hybrid) @@ -661,55 +340,139 @@ stemming) and searched **twice** against Postgres: The two ranked lists are fused with **Reciprocal Rank Fusion** (`score = Ξ£ 1/(k + rank)`, `BOR_RRF_K=60`) β€” a chunk in both lists scores -nearly double, which is what lets a name-your-tool question ("gitlab") find -its own document even when the question embeds close to generic templates. +nearly double, which lets a name-your-tool question ("gitlab") find its own +document even when the question embeds close to generic templates. The **honesty gate** (A8) then answers (HIGH) when the best cosine is β‰₯ `BOR_RELEVANCE_THRESHOLD` (default `0.62`) **or** at least one chunk matched lexically (`fts_hits > 0`) β€” it deflects (LOW) only when *both* signals are -absent. The top `BOR_TOP_N_DOCS` full documents are still what the LLM sees. +absent. The top `BOR_TOP_N_DOCS` full documents are what the LLM sees. -`query_log` records every turn (`top_score` = best cosine, `fts_hits`, -`chunk_hits`, `deflected`, `sources`, `latency_ms`) β€” the raw material for -tuning: `psql … -c 'SELECT question, top_score, fts_hits, deflected FROM query_log ORDER BY created_at DESC LIMIT 20'`. +--- ## Document summaries (non-markdown) Raw yaml/json/py/txt embeds badly β€” flags and keys are not language, so retrieval can miss exactly the documents that are all configuration. At import time, every **non-markdown** A9 document is summarized by the aipi -`lite` model (one-shot `LLMClient.chat`, `BOR_LLM_SUMMARY_MODEL`, default -`lite`): +`lite` model (`BOR_LLM_SUMMARY_MODEL`, default `lite`): -* the summary is stored on `documents.summary` **and** indexed as one - extra embedded chunk (`chunks.is_summary`, position βˆ’1), so hybrid - search has a natural-language target to hit instead of the raw text; -* the last line is a **code-deterministic** pointer β€” `Source: - /` β€” appended by the app, never model-generated; -* the model only sees the first `BOR_SUMMARY_MAX_CHARS` (default 12000) - characters of the document; overflow is cut and marked with the shared - `[…truncated…]` marker. +* The summary is stored on `documents.summary` **and** indexed as one extra + embedded chunk (`chunks.is_summary`, position βˆ’1), so hybrid search has a + natural-language target to hit instead of the raw text. +* The last line is a **code-deterministic** pointer β€” `Source: /` + β€” appended by the app, never model-generated. +* The model only sees the first `BOR_SUMMARY_MAX_CHARS` (default 12000) + characters; overflow is cut and marked with `[…truncated…]`. -A summary hit resolves through the normal chunkβ†’document mapping: the -LLM receives the **full source document** (never the summary alone, -never truncated). Markdown files are natural language already and get no -summary. Summary generation is best-effort: if `lite` fails for a file, -the document is still indexed (without a summary), the failure is logged -and counted in the import summary line (`summaries=N summary_errors=N`). -On a chat turn, the per-turn log line records `summary_hits=N` β€” how many -summary chunks of the selected context the question landed on. +Summary generation is best-effort: if `lite` fails, the document is still +indexed (without a summary), the failure is logged, and counted in the +import summary line (`summaries=N summary_errors=N`). + +--- + +## Caching & deploys + +A deploy is a commit β€” and the browser must see it **without a hard refresh**. +One Starlette middleware (`app/core/caching.py`) applies the rule at the +transport layer: + +- **HTML pages** ship `Cache-Control: no-cache`, **no `etag`**, **no + `last-modified`** β€” each visit always gets a fresh 200 body. +- **Assets are versioned and cached for a year.** Pages reference their + CSS/JS with a token (`/assets/styles.css?v=`), and every + `/assets/*` response ships `Cache-Control: public, max-age=31536000, immutable`. +- **The token is the deploy.** In a git checkout it is the short SHA of + `HEAD` β€” so every commit/deploy flips the token and the versioned asset + URLs change with it. A checkout without `.git` falls back to a stable + content hash of the `frontend/` tree. + +No CDN, no new services, no build-step change: the middleware rewrites the +asset references of the known pages in flight. + +> **Deploy note:** the very first deploy onto this scheme needs one normal +> page visit, so the browser revalidates the HTML once and starts requesting +> the versioned assets; every commit after that is picked up automatically. + +--- + +## Configuration reference + +| Env | Default | Meaning | +|-----|---------|---------| +| `BOR_APP_NAME` | `Brain of Reese` | Display name everywhere (page titles, header brand, status labels) | +| `BOR_INPUT_PLACEHOLDER` | `Ask me anything…` | Chat composer placeholder | +| `BOR_FOOTER_TEXT` | `Powered by self-hosted models` | Footer line on every page | +| `BOR_THEME` | *(empty)* | Filename under `frontend/assets/themes/` (e.g. `indigo.css`) β€” a `:root` palette override | +| `BOR_DATABASE_URL` | local compose URL | SQLAlchemy URL (psycopg) | +| `BOR_LLM_BASE_URL` | `https://aipi.reeseapps.com/v1` | OpenAI-compatible endpoint | +| `BOR_LLM_API_KEY` | β€” (falls back to `$AIPI_KEY`) | aipi API key | +| `BOR_LLM_CHAT_MODEL` | `turbo` | Chat model | +| `BOR_LLM_EMBED_MODEL` | `embed` | Embedding model | +| `BOR_LLM_SUMMARY_MODEL` | `lite` | One-shot completions: document summaries at import, KB overview | +| `BOR_EMBEDDING_DIM` | `768` | Vector dimension (fixed at table creation) | +| `BOR_TOP_N_DOCS` | `2` | Full documents fed to the LLM | +| `BOR_RELEVANCE_THRESHOLD` | `0.62` | Answer when best cosine β‰₯ this **or** an FTS hit; below + no FTS β‡’ honest deflection | +| `BOR_HYBRID_VECTOR_CANDIDATES` | `100` | Cosine list width for the RRF fusion | +| `BOR_HYBRID_LEXICAL_CANDIDATES` | `30` | FTS list width for the RRF fusion | +| `BOR_RRF_K` | `60` | RRF damping constant (`1/(k + rank)`) | +| `BOR_AGENT_MAX_ROUNDS` | `10` | Hard cap on agent tool rounds per grounded turn (0 = no tools) | +| `BOR_IMPORT_EXTENSIONS` | csv (see below) | Importable formats (may only narrow the A9 set) | +| `BOR_GIT_SOURCES` | β€” (empty) | CSV of git repo URLs β€” **fallback while the admin Git sources page's list is empty** | +| `BOR_SOURCES_DIR` | `~/bor-sources` | Where git repos are cloned/pulled | +| `BOR_UPLOAD_DIR` | `~/bor-sources/uploads` | Where uploaded source archives are unpacked | +| `BOR_UPLOAD_MAX_MB` | `512` | Cap (MiB) for uploaded source archives (zip-bomb guard) | +| `BOR_STEERING_MAX_CHARS` | `8000` | Char budget for the `` prompt section | +| `BOR_SUMMARY_MAX_CHARS` | `12000` | Cap on document content sent to the `lite` summary model | +| `BOR_KB_OVERVIEW_MAX_CHARS` | `4000` | Char budget for the `` prompt section | +| `BOR_OVERVIEW_INPUT_MAX_CHARS` | `40000` | Cap on the document list sent to `lite` for KB overview generation | +| `BOR_SUGGESTIONS` | built-in list | JSON seed for onboarding chips (shown before the first saved question) | +| `BOR_ADMIN_PASSWORD` | *(required)* | The single admin's password β€” app refuses to start when empty | +| `BOR_SESSION_SECRET` | *(required)* | Signing key for the `bor_session` cookie | +| `BOR_SESSION_MAX_AGE` | `43200` | Session-cookie lifetime in seconds (12 h, sliding) | +| `DEBUGPY` | `0` | `1` β‡’ attach-on-demand debugpy on `DEBUGPY_PORT` (default 5678) | +| `BOR_LOG_LEVEL` | `INFO` | App log level | + +### Customizing the look + +Every identity string is an env var: `BOR_APP_NAME` (display name), +`BOR_INPUT_PLACEHOLDER` (chat composer placeholder), and +`BOR_FOOTER_TEXT` (footer line) β€” all served by `GET /api/config` and +applied by `assets/brand.js` at boot. + +Color themes are plain CSS variable overrides: write a `:root` block in +`frontend/assets/themes/` and point `BOR_THEME` at the filename. The server +refuses a malformed `BOR_THEME` at startup; a missing file degrades to the +built-in palette. Leave everything unset and the app renders the defaults +byte-identically. + +--- + +## Checking retrieval quality + +Ask the *real* pipeline (live aipi embeddings + the current KB) whether a +question lands on the right document, with gate verdict and per-document +cosine / FTS / fused scores: + +```bash +uv run python -m scripts.eval_retrieval "How did I install gitlab?" +uv run python -m scripts.eval_retrieval --from-file questions.txt --top 8 +``` + +Requires `AIPI_KEY` in the environment and an imported knowledge base. + +--- ## Debugging -`debugpy` is **off by default** and *never imported* unless you opt in β€” -zero overhead in normal runs. +`debugpy` is **off by default** and *never imported* unless you opt in: ```bash DEBUGPY=1 uv run uvicorn app.main:app # β†’ log line: debugpy: remote debugging ENABLED, listening on 0.0.0.0:5678 ``` -Then attach from VS Code (`.vscode/launch.json`): +Attach from VS Code (`.vscode/launch.json`): ```json { "name": "Attach to Brain of Reese", @@ -721,10 +484,13 @@ Then attach from VS Code (`.vscode/launch.json`): ] } ``` + The port is non-blocking and attach-on-demand: the app keeps running normally until you attach. Override the port with `DEBUGPY_PORT`. -## QA / Testing Environment +--- + +## QA / Testing Three layers β€” the project rule is **one story, one phase, one Playwright suite** (see `AGENTS.md`): @@ -733,7 +499,7 @@ suite** (see `AGENTS.md`): # Unit + integration (FastAPI TestClient) uv run pytest -# Same, with the coverage gate (phases require >90% on app/) +# Coverage gate (phases require >90% on app/) uv run pytest --cov=app --cov-report=term-missing # Lint + static types @@ -743,23 +509,22 @@ uv run pyright # Playwright E2E β€” install the browser once: uv run playwright install chromium -# Each story's E2E runs IN ISOLATION (DB must be up): -podman compose up -d db +# Each story's E2E runs IN ISOLATION: uv run pytest tests/e2e/test_import_documents.py -v --no-cov uv run pytest tests/e2e/test_chat_rag.py -v --no-cov # ...one file per story in .agents/user_stories/ (see .agents/phases/todo/) ``` **Deterministic E2E:** by default the E2E app talks to a local **mock -aipi** (`tests/e2e/mock_llm.py`) whose embeddings are real -token-overlap vectors β€” so the cosine relevance threshold behaves like -production (on-topic questions answer, off-topic ones deflect). -To run E2E against the **live** self-hosted models instead: +aipi** (`tests/e2e/mock_llm.py`) whose embeddings are real token-overlap +vectors β€” so the cosine relevance threshold behaves like production. To +run E2E against the **live** self-hosted models instead: ```bash E2E_REAL_LLM=1 uv run pytest tests/e2e/test_chat_rag.py -v --no-cov ``` -(requires a real import of your docs first). + +--- ## Production Deployment @@ -779,6 +544,7 @@ podman run -d --name brain-of-reese \ -e BOR_LLM_API_KEY=$AIPI_KEY \ brain-of-reese/app:latest ``` + The entrypoint runs `alembic upgrade head` automatically on start. Or run the whole stack from compose (app + db): @@ -786,99 +552,52 @@ Or run the whole stack from compose (app + db): podman compose --profile prod up -d --build ``` -Production hardening notes: app runs as non-root (uid 10001), slim image, +Production hardening: app runs as non-root (uid 10001), slim image, healthcheck on `/api/health`, debugpy off unless `DEBUGPY=1`, all assets served locally (no CDN), `BOR_ENVIRONMENT=production`. -## Configuration reference - -| Env | Default | Meaning | -|-----|---------|---------| -| `BOR_APP_NAME` | `Brain of Reese` | the display name everywhere (phase 39): every page ``, the header brand, the chat status labels ("… is thinking"), the empty-state greeting, and the aria/placeholder text. Served to the frontend by `GET /api/config` and applied by `assets/brand.js`; a name starting `Brain of ` keeps the bold split (`Brain of <strong>rest</strong>`), any other name renders in normal weight. Unset β‡’ byte-identical to the default | -| `BOR_INPUT_PLACEHOLDER` | `Ask me anything…` | the chat composer placeholder (`#message-input`, chat page only); applied by `assets/brand.js` from `GET /api/config`; unset β‡’ the template default | -| `BOR_FOOTER_TEXT` | `Powered by self-hosted models` | the footer line on all 9 pages (the `.footer-text` spans); same mechanism; unset β‡’ the template default | -| `BOR_THEME` | *(empty)* | a filename under `frontend/assets/themes/` (e.g. `indigo.css`) β€” a `:root` palette override injected after `styles.css` (later wins the cascade); the server refuses a malformed name at startup (bare `^[a-z0-9_-]+\.css$` filename); a missing file degrades to the built-in theme; unset β‡’ the built-in dark-tech palette | -| `BOR_DATABASE_URL` | local compose URL | SQLAlchemy URL (psycopg) | -| `BOR_LLM_BASE_URL` | `https://aipi.reeseapps.com/v1` | OpenAI-compatible endpoint | -| `BOR_LLM_API_KEY` | β€” (falls back to `$AIPI_KEY`) | aipi API key | -| `BOR_LLM_CHAT_MODEL` | `turbo` | chat model | -| `BOR_LLM_EMBED_MODEL` | `embed` | embedding model | -| `BOR_LLM_SUMMARY_MODEL` | `lite` | one-shot (non-streaming) completions: document summaries at import (phase 30) and the KB overview (phase 31) | -| `BOR_EMBEDDING_DIM` | `768` | vector dimension (fixed at table creation) | -| `BOR_TOP_N_DOCS` | `2` | full documents fed to the LLM | -| `BOR_RELEVANCE_THRESHOLD` | `0.62` | answer when best cosine β‰₯ this **or** an FTS hit; below + no FTS β‡’ honest deflection | -| `BOR_HYBRID_VECTOR_CANDIDATES` | `100` | cosine list width for the RRF fusion | -| `BOR_HYBRID_LEXICAL_CANDIDATES` | `30` | FTS list width for the RRF fusion | -| `BOR_RRF_K` | `60` | RRF damping constant (`1/(k + rank)`) | -| `BOR_AGENT_MAX_ROUNDS` | `10` | hard cap on agent tool rounds per grounded turn β€” every call the model emits consumes a round; at the cap the loop forces one final no-tools answer (0 = no tools, the kill switch) | -| `BOR_IMPORT_EXTENSIONS` | `md,markdown,txt,yaml,yml,json,py,container,network,volume,image,pod,kube,swap,os,endpoint,j2` | csv of importable formats (may only narrow the A9 set) | -| `BOR_GIT_SOURCES` | β€” (empty) | csv of git repo URLs β€” **fallback while the admin Git sources page's list (Postgres `git_sources`) is empty**; the page is the primary management surface (see *Git-based sources*). **Git-only**: local directory sources have no env var β€” they are registered on the admin page (see *Local directory sources*) | -| `BOR_SOURCES_DIR` | `~/bor-sources` | where the git source repos are cloned/pulled (one subdirectory per repo) | -| `BOR_UPLOAD_DIR` | `~/bor-sources/uploads` | where uploaded source archives are unpacked β€” one subdirectory per source name (filename minus the archive suffix); separate from the git checkouts (see *Archive upload sources*) | -| `BOR_UPLOAD_MAX_MB` | `512` | cap (MiB) for uploaded source archives β€” bounds **both** the compressed upload and the total extracted bytes (zip-bomb guard); must be > 0 | -| `BOR_STEERING_MAX_CHARS` | `8000` | char budget for the `<tuning>` (steering notes) prompt section | -| `BOR_SUMMARY_MAX_CHARS` | `12000` | cap on document content sent to the `lite` summary model at import (see *Document summaries*) | -| `BOR_KB_OVERVIEW_MAX_CHARS` | `4000` | char budget for the `<knowledge_base>` (KB overview) prompt section | -| `BOR_OVERVIEW_INPUT_MAX_CHARS` | `40000` | cap on the document list sent to the `lite` model when generating the KB overview | -| `BOR_SUGGESTIONS` | built-in list | JSON seed for the onboarding chips β€” shown only before the first saved question; afterwards the chips are the last 3 questions asked (phase 80) | -| `BOR_ADMIN_PASSWORD` | *(required)* | the single admin's password (plaintext, `.env`); app refuses to start when empty | -| `BOR_SESSION_SECRET` | *(required)* | signing key for the `bor_session` cookie; `python -c 'import secrets;print(secrets.token_hex(32))'` | -| `BOR_SESSION_MAX_AGE` | `43200` | session-cookie lifetime in seconds (12 h, sliding) | -| `DEBUGPY` | `0` | `1` β‡’ attach-on-demand debugpy on `DEBUGPY_PORT` (default 5678) | -| `BOR_LOG_LEVEL` | `INFO` | app log level | - -### Customizing the look - -The app ships as β€œBrain of Reese”, but every identity string is an env -var: `BOR_APP_NAME` (display name), `BOR_INPUT_PLACEHOLDER` (chat composer -placeholder), and `BOR_FOOTER_TEXT` (the footer line on every page) β€” all -served by `GET /api/config` and applied by `assets/brand.js` at boot. -Color themes are plain CSS variable overrides: write a `:root` block in -`frontend/assets/themes/` and point `BOR_THEME` at the filename -([the themes `README.md`](frontend/assets/themes/README.md) is the -authoring guide, `indigo.css` the working example). The server refuses a -malformed `BOR_THEME` at startup, and a missing theme file degrades to the -built-in palette β€” the page never breaks. Leave everything unset and the -app renders the defaults byte-identically: the dark-tech palette shown -throughout this README is the no-config default. +--- ## Troubleshooting - **`401` from aipi** β€” set `BOR_LLM_API_KEY` (or `$AIPI_KEY`). -- **`litellm.UnsupportedParamsError … encoding_format` from aipi** β€” the - aipi proxy (litellm `openai_like`) rejects the `encoding_format` parameter - that the `openai` SDK injects into every embeddings request. The app - already works around this by POSTing a minimal `{model, input}` payload - through the openai client's own httpx transport (`app/rag/llm.py` β†’ - `LLMClient._embed_batch`). If you see this, you are likely calling the - endpoint with a different client β€” drop the parameter (or set - `litellm.drop_params = True` on the proxy). +- **`litellm.UnsupportedParamsError … encoding_format`** β€” the aipi proxy + (litellm `openai_like`) rejects the `encoding_format` parameter. The app + already works around this by POSTing a minimal `{model, input}` payload. + If you see this, you are likely calling the endpoint with a different + client β€” drop the parameter (or set `litellm.drop_params = True` on the proxy). - **Embedding dimension mismatch** β€” aipi changed models; run `uv run python -m scripts.llm_probe`, update `BOR_EMBEDDING_DIM`, then - drop + recreate the chunks table (new migration or manual `TRUNCATE - chunks, documents`). -- **Honest deflection (the amber β€œI haven't done anything like that” - bubble)** β€” every question passes the honesty gate: deflection happens - only when the best cosine similarity is below - `BOR_RELEVANCE_THRESHOLD` (default `0.62`) **and** no chunk matched the - question lexically (`fts_hits = 0`). A weak cosine with a lexical hit - (name-your-tool questions) still gets a grounded answer. When it does - deflect, the LLM prompt carries weak-hit *titles only* (no document - content), the reply opens with β€œI haven't done anything like that”, the - bubble renders amber with β€œMaybe try” chips derived from the closest - indexed titles, the SSE `done` event carries `deflected: true` + - `suggestions[]`, and the `query_log` row records `deflected=true` + the - weak `top_score` + `fts_hits`. This is a feature, not a bug β€” the KB - simply has no notes that close; the chips always point at topics Brain - really covers. + drop + recreate the chunks table. +- **Honest deflection** (the amber *"I haven't done anything like that"* + bubble) β€” every question passes the honesty gate: deflection happens only + when the best cosine similarity is below `BOR_RELEVANCE_THRESHOLD` + (default `0.62`) **and** no chunk matched lexically (`fts_hits = 0`). + A weak cosine with a lexical hit (name-your-tool questions) still gets a + grounded answer. When it does deflect, the LLM prompt carries weak-hit + *titles only* (no document content), the reply opens with *"I haven't + done anything like that"*, the bubble renders amber with *"Maybe try"* + chips derived from the closest indexed titles, and the `query_log` row + records `deflected=true`. This is a feature, not a bug. - **Answers deflect too often / too rarely** β€” tune `BOR_RELEVANCE_THRESHOLD` (lower = answers more, higher = more honest deflection): `0.0` β‡’ the gate leans entirely on FTS hits; `1.0` β‡’ everything deflects unless a chunk matches lexically. The `embed` model's cosines cluster in a ~0.6–0.85 band on the live KB, so the default is - `0.62`; after changing it, check the real scores: - `psql … -c 'SELECT question, top_score, fts_hits, deflected FROM query_log ORDER BY created_at DESC LIMIT 20'` + `0.62`. Check real scores: + ```sql + SELECT question, top_score, fts_hits, deflected + FROM query_log ORDER BY created_at DESC LIMIT 20; + ``` - **KB offline banner in the chat** β€” Postgres isn't running: `podman compose up -d db`. - **Stuck "Thinking…"** β€” the LLM is slow or down; a 120s client timeout turns it into an error banner automatically. + +--- + +## Planning & Architecture + +The architecture, LOCKED decisions, and the phase roadmap live in +[`.agents/PLAN.md`](.agents/PLAN.md); per-story specs in +[`.agents/user_stories/`](.agents/user_stories/).