Files

T

ducoterra 022da8e2bc feat: scaffold Brain of Reese — FastAPI RAG chat over Postgres 17 + pgvector

Foundation (phase 01, verified):
- FastAPI app: /api/health, /api/suggestions, /api/chat (placeholder),
  static frontend served locally (no CDN)
- Postgres 17 + pgvector via db/Containerfile + compose.yaml
  (podman compose up -d db), Alembic initial migration (documents,
  chunks with vector(768), query_log)
- LLM client targeting https://aipi.reeseapps.com/v1 (turbo/embed);
  scripts/llm_probe.py verified models + 768-dim embeddings live
- Conditional debugpy: imported only when DEBUGPY=1 (attach on demand,
  :5678); logging config for clean single-line logs
- Frontend shell: mobile-first chat + Sources pages, tokens, a11y baselines
- Tests: 24 unit+integration (99% coverage on app/), ruff + pyright clean,
  Playwright smoke E2E (3 tests) against a deterministic mock LLM
- Planning: .agent/PLAN.md (architecture + LOCKED decisions), AGENTS.md,
  6 user stories, 7 phase files (one story / one phase / one Playwright
  suite each)

2026-08-21 13:42:21 -04:00

3.4 KiB

Raw Blame History

Story: Import Documents

Phase: 02_story_import_documents.md · E2E: tests/e2e/test_import_documents.py

Narrative

As Reese (the owner), I want to point the importer at one or more directories of markdown files and have them chunked, embedded, and stored in Postgres — so that Brain's answers always reflect my current documentation.

Given the ~/Homelab and ~/Deployments trees (or any --source dirs)
When I run uv run python -m scripts.import_docs
Then every *.md file (after the exclusion list) is present in the documents table with its full content, a sha256 hash, and chunk rows with 768-dim embeddings; unchanged files are skipped on re-runs; and the Sources page in the browser shows the indexed documents.

Acceptance criteria

scripts/import_docs.py accepts repeatable --source PATH (default ~/Homelab ~/Deployments), --prune, and --limit N (debug).
Only *.md files are imported; excluded dirs: .venv, node_modules, .git, __pycache__, .pytest_cache, dist, build (PLAN A9).
Delta detection by sha256 on (source, path): unchanged → skipped (no re-embedding); changed → re-chunked + re-embedded, old chunks replaced atomically.
Embeddings are batched (BOR_EMBED_BATCH_SIZE) against aipi /v1/embeddings (embed); a dimension mismatch fails loudly with an actionable message.
Rich per-file logging (added|updated|unchanged|pruned) + summary.
GET /api/docs returns the document list; the Sources page renders it (stat cards + table) or the designed empty state when none exist.
The whole flow works against the mock LLM in E2E (deterministic), and against real aipi for manual runs.

UI Visualization & Structure

Sources page (/sources.html), desktop: header row (h1 + sub), then stat cards in repeat(auto-fit, minmax(170px,1fr)) (documents / chunks / last indexed), then a full-width table inside a scroll wrapper (min-width 640px → horizontal scroll, never a squeezed hairline list). Columns: Source · Path (mono, ellipsized w/ title) · Title · Chunks · Indexed. Uses ≥85% of the 72rem container width.
Empty state (no docs): centered card with 📂, "Nothing indexed yet", and the exact import command in a <code> pill. No dead links, no placeholder tables.
Accessibility: <caption class="visually-hidden"> on the table, scope="col" on headers, role="region" + tabindex="0" on the scroll wrapper (keyboard scrollable), stat values have visible labels.
Mobile: stat cards stack (auto-fit), table scrolls horizontally, no content below the fold is unreachable.

Playwright Mapping Rule

Test Scenario → tests/e2e/test_import_documents.py (one isolated Playwright suite for this story):

Seeding: run the import function in-process against tests/fixtures/docs/ (mock embeddings, temp DB state) — a fixture, not the test's subject.
test_sources_page_lists_indexed_docs — goto /sources.html, assert stat cards show the fixture counts and the table rows include homelab/kubernetes.md, homelab/backups.md, deployments/new-service.md.
test_sources_table_layout — table wrapper width ≥80% of container; caption present; on a 375px viewport the wrapper scrolls horizontally.
test_empty_state_when_no_docs (fresh/truncated DB) — empty state visible with the import command; table hidden.

3.4 KiB Raw Blame History