feat: scaffold Brain of Reese — FastAPI RAG chat over Postgres 17 + pgvector
Foundation (phase 01, verified): - FastAPI app: /api/health, /api/suggestions, /api/chat (placeholder), static frontend served locally (no CDN) - Postgres 17 + pgvector via db/Containerfile + compose.yaml (podman compose up -d db), Alembic initial migration (documents, chunks with vector(768), query_log) - LLM client targeting https://aipi.reeseapps.com/v1 (turbo/embed); scripts/llm_probe.py verified models + 768-dim embeddings live - Conditional debugpy: imported only when DEBUGPY=1 (attach on demand, :5678); logging config for clean single-line logs - Frontend shell: mobile-first chat + Sources pages, tokens, a11y baselines - Tests: 24 unit+integration (99% coverage on app/), ruff + pyright clean, Playwright smoke E2E (3 tests) against a deterministic mock LLM - Planning: .agent/PLAN.md (architecture + LOCKED decisions), AGENTS.md, 6 user stories, 7 phase files (one story / one phase / one Playwright suite each)
This commit is contained in:
@@ -0,0 +1,62 @@
|
||||
# Story: Import Documents
|
||||
|
||||
**Phase:** `02_story_import_documents.md` · **E2E:** `tests/e2e/test_import_documents.py`
|
||||
|
||||
## Narrative
|
||||
|
||||
As **Reese** (the owner), I want to point the importer at one or more
|
||||
directories of markdown files and have them chunked, embedded, and stored in
|
||||
Postgres — so that Brain's answers always reflect my *current* documentation.
|
||||
|
||||
- **Given** the `~/Homelab` and `~/Deployments` trees (or any `--source` dirs)
|
||||
- **When** I run `uv run python -m scripts.import_docs`
|
||||
- **Then** every `*.md` file (after the exclusion list) is present in the
|
||||
`documents` table with its full content, a sha256 hash, and chunk rows with
|
||||
768-dim embeddings; unchanged files are skipped on re-runs; and the
|
||||
Sources page in the browser shows the indexed documents.
|
||||
|
||||
## Acceptance criteria
|
||||
1. `scripts/import_docs.py` accepts repeatable `--source PATH` (default
|
||||
`~/Homelab` `~/Deployments`), `--prune`, and `--limit N` (debug).
|
||||
2. Only `*.md` files are imported; excluded dirs: `.venv`, `node_modules`,
|
||||
`.git`, `__pycache__`, `.pytest_cache`, `dist`, `build` (PLAN A9).
|
||||
3. Delta detection by sha256 on `(source, path)`: unchanged → skipped
|
||||
(no re-embedding); changed → re-chunked + re-embedded, old chunks
|
||||
replaced atomically.
|
||||
4. Embeddings are batched (`BOR_EMBED_BATCH_SIZE`) against `aipi /v1/embeddings`
|
||||
(`embed`); a dimension mismatch fails loudly with an actionable message.
|
||||
5. Rich per-file logging (`added|updated|unchanged|pruned`) + summary.
|
||||
6. `GET /api/docs` returns the document list; the Sources page renders it
|
||||
(stat cards + table) or the designed empty state when none exist.
|
||||
7. The whole flow works against the **mock LLM** in E2E (deterministic),
|
||||
and against real aipi for manual runs.
|
||||
|
||||
## UI Visualization & Structure
|
||||
- **Sources page (`/sources.html`), desktop:** header row (h1 + sub), then
|
||||
stat cards in `repeat(auto-fit, minmax(170px,1fr))` (documents / chunks /
|
||||
last indexed), then a **full-width table** inside a scroll wrapper
|
||||
(min-width 640px → horizontal scroll, never a squeezed hairline list).
|
||||
Columns: Source · Path (mono, ellipsized w/ `title`) · Title · Chunks ·
|
||||
Indexed. Uses ≥85% of the 72rem container width.
|
||||
- **Empty state (no docs):** centered card with 📂, "Nothing indexed yet",
|
||||
and the exact import command in a `<code>` pill. No dead links, no
|
||||
placeholder tables.
|
||||
- **Accessibility:** `<caption class="visually-hidden">` on the table,
|
||||
`scope="col"` on headers, `role="region"` + `tabindex="0"` on the scroll
|
||||
wrapper (keyboard scrollable), stat values have visible labels.
|
||||
- **Mobile:** stat cards stack (auto-fit), table scrolls horizontally,
|
||||
no content below the fold is unreachable.
|
||||
|
||||
## Playwright Mapping Rule
|
||||
**Test Scenario → `tests/e2e/test_import_documents.py`** (one isolated
|
||||
Playwright suite for this story):
|
||||
1. *Seeding:* run the import function in-process against
|
||||
`tests/fixtures/docs/` (mock embeddings, temp DB state) — a fixture, not
|
||||
the test's subject.
|
||||
2. `test_sources_page_lists_indexed_docs` — goto `/sources.html`, assert stat
|
||||
cards show the fixture counts and the table rows include
|
||||
`homelab/kubernetes.md`, `homelab/backups.md`, `deployments/new-service.md`.
|
||||
3. `test_sources_table_layout` — table wrapper width ≥80% of container;
|
||||
`caption` present; on a 375px viewport the wrapper scrolls horizontally.
|
||||
4. `test_empty_state_when_no_docs` (fresh/truncated DB) — empty state visible
|
||||
with the import command; table hidden.
|
||||
Reference in New Issue
Block a user