feat: scaffold Brain of Reese — FastAPI RAG chat over Postgres 17 + pgvector

Foundation (phase 01, verified):
- FastAPI app: /api/health, /api/suggestions, /api/chat (placeholder),
  static frontend served locally (no CDN)
- Postgres 17 + pgvector via db/Containerfile + compose.yaml
  (podman compose up -d db), Alembic initial migration (documents,
  chunks with vector(768), query_log)
- LLM client targeting https://aipi.reeseapps.com/v1 (turbo/embed);
  scripts/llm_probe.py verified models + 768-dim embeddings live
- Conditional debugpy: imported only when DEBUGPY=1 (attach on demand,
  :5678); logging config for clean single-line logs
- Frontend shell: mobile-first chat + Sources pages, tokens, a11y baselines
- Tests: 24 unit+integration (99% coverage on app/), ruff + pyright clean,
  Playwright smoke E2E (3 tests) against a deterministic mock LLM
- Planning: .agent/PLAN.md (architecture + LOCKED decisions), AGENTS.md,
  6 user stories, 7 phase files (one story / one phase / one Playwright
  suite each)
This commit is contained in:
2026-08-21 13:42:21 -04:00
commit 022da8e2bc
63 changed files with 5225 additions and 0 deletions
+62
View File
@@ -0,0 +1,62 @@
# Story: Import Documents
**Phase:** `02_story_import_documents.md` · **E2E:** `tests/e2e/test_import_documents.py`
## Narrative
As **Reese** (the owner), I want to point the importer at one or more
directories of markdown files and have them chunked, embedded, and stored in
Postgres — so that Brain's answers always reflect my *current* documentation.
- **Given** the `~/Homelab` and `~/Deployments` trees (or any `--source` dirs)
- **When** I run `uv run python -m scripts.import_docs`
- **Then** every `*.md` file (after the exclusion list) is present in the
`documents` table with its full content, a sha256 hash, and chunk rows with
768-dim embeddings; unchanged files are skipped on re-runs; and the
Sources page in the browser shows the indexed documents.
## Acceptance criteria
1. `scripts/import_docs.py` accepts repeatable `--source PATH` (default
`~/Homelab` `~/Deployments`), `--prune`, and `--limit N` (debug).
2. Only `*.md` files are imported; excluded dirs: `.venv`, `node_modules`,
`.git`, `__pycache__`, `.pytest_cache`, `dist`, `build` (PLAN A9).
3. Delta detection by sha256 on `(source, path)`: unchanged → skipped
(no re-embedding); changed → re-chunked + re-embedded, old chunks
replaced atomically.
4. Embeddings are batched (`BOR_EMBED_BATCH_SIZE`) against `aipi /v1/embeddings`
(`embed`); a dimension mismatch fails loudly with an actionable message.
5. Rich per-file logging (`added|updated|unchanged|pruned`) + summary.
6. `GET /api/docs` returns the document list; the Sources page renders it
(stat cards + table) or the designed empty state when none exist.
7. The whole flow works against the **mock LLM** in E2E (deterministic),
and against real aipi for manual runs.
## UI Visualization & Structure
- **Sources page (`/sources.html`), desktop:** header row (h1 + sub), then
stat cards in `repeat(auto-fit, minmax(170px,1fr))` (documents / chunks /
last indexed), then a **full-width table** inside a scroll wrapper
(min-width 640px → horizontal scroll, never a squeezed hairline list).
Columns: Source · Path (mono, ellipsized w/ `title`) · Title · Chunks ·
Indexed. Uses ≥85% of the 72rem container width.
- **Empty state (no docs):** centered card with 📂, "Nothing indexed yet",
and the exact import command in a `<code>` pill. No dead links, no
placeholder tables.
- **Accessibility:** `<caption class="visually-hidden">` on the table,
`scope="col"` on headers, `role="region"` + `tabindex="0"` on the scroll
wrapper (keyboard scrollable), stat values have visible labels.
- **Mobile:** stat cards stack (auto-fit), table scrolls horizontally,
no content below the fold is unreachable.
## Playwright Mapping Rule
**Test Scenario → `tests/e2e/test_import_documents.py`** (one isolated
Playwright suite for this story):
1. *Seeding:* run the import function in-process against
`tests/fixtures/docs/` (mock embeddings, temp DB state) — a fixture, not
the test's subject.
2. `test_sources_page_lists_indexed_docs` — goto `/sources.html`, assert stat
cards show the fixture counts and the table rows include
`homelab/kubernetes.md`, `homelab/backups.md`, `deployments/new-service.md`.
3. `test_sources_table_layout` — table wrapper width ≥80% of container;
`caption` present; on a 375px viewport the wrapper scrolls horizontally.
4. `test_empty_state_when_no_docs` (fresh/truncated DB) — empty state visible
with the import command; table hidden.