feat(rag): git-based import sources — BOR_GIT_SOURCES repos cloned (first run, --depth 1) or pulled (--ff-only) into BOR_SOURCES_DIR/<repo>/ then indexed; --source still wins; a failed sync aborts before importing anything

README.md

@@ -60,9 +60,22 @@ uv run alembic upgrade head
 ### 5. Import your knowledge base
 ```bash
 uv run python -m scripts.llm_probe      # sanity: models + 768-dim check
-uv run python -m scripts.import_docs    # defaults: ~/Homelab + ~/Deployments
+uv run python -m scripts.import_docs    # import the configured sources (below)
 ```
 
+Two source modes:
+
+- **Git sources (recommended)** — set `BOR_GIT_SOURCES` in `.env` to a
+  comma-separated list of git repo URLs. `import_docs` clones each repo
+  (first run) or pulls it (subsequent runs) into
+  `BOR_SOURCES_DIR/<repo-name>/` (default `~/bor-sources`) and indexes the
+  checkouts — see [Git-based sources](#git-based-sources).
+- **Manual directories** — `--source <path>` (repeatable) imports local
+  directories directly and *always wins* over `BOR_GIT_SOURCES`.
+- If neither is set, the import falls back to the **previous** default,
+  `~/Homelab` + `~/Deployments` — kept only for backwards compatibility,
+  now replaced by `BOR_GIT_SOURCES`.
+
 ### 6. Run the app
 ```bash
 uv run uvicorn app.main:app --reload
@@ -76,16 +89,19 @@ uv run uvicorn app.main:app --reload
 
 - **Chat** (`/`) — ask questions; answers stream in with **source chips**
   that cite the exact documents used. Clicking a chip opens that document
-  **in a new tab**.
-- **Document viewer** (`/document.html?source=…&path=…`) — the full text of
-  any indexed document, served from the database (no filesystem access):
+  in an **almost-fullscreen modal** on the same page (no new tab).
+- **Document viewer** — the modal above *is* the viewer; the full text of
+  any indexed document is served from the database (no filesystem access):
   markdown is rendered, every other format (`yaml`, `json`, `py`, `txt`, …)
-  is shown as escaped monospace text. Unknown documents get a designed
-  not-found state with a link back to the index.
+  is shown as escaped monospace text. `/document.html?source=…&path=…`
+  stays as the **full-page / direct-link** form (the modal's “Full page”
+  button and the URL to share — it works without JS). Unknown documents
+  get a designed not-found state with a link back to the index.
 - **Sources** (`/sources.html`) — the indexed document list; the *Path*
-  column links each document to the viewer in a new tab. **Admin-only** —
-  anonymous visitors see a sign-in gate instead (the catalog is what the
-  login locks; the document viewer itself stays open to everyone).
+  column opens each document in the same **almost-fullscreen modal** (no
+  new tab). **Admin-only** — anonymous visitors see a sign-in gate instead
+  (the catalog is what the login locks; the document viewer itself stays
+  open to everyone).
 
 ## Thinking
 
@@ -205,6 +221,10 @@ uv run python -m scripts.import_docs --prune         # also drop deleted/out-of-
 uv run python -m scripts.import_docs --source ~/SomeOtherDocs
 ```
 
+With **git-based sources** (below) each run first pulls the latest commits
+of your repos, so this same command is the whole update loop: commit in the
+repo → re-run the import.
+
 Then check the **Sources** page (`http://localhost:8000/sources.html`):
 the *documents* / *chunks* counters and *last indexed* timestamp should
 reflect the new files, and each document row shows when it was last
@@ -230,6 +250,35 @@ embedded.
 - To sanity-check the LLM backend (models + embedding dimension) after any
   aipi change: `uv run python -m scripts.llm_probe`.
 
+### Git-based sources
+
+Rather than pointing the import at local folders, point it at **git
+repositories** — the notes live in the repos and `import_docs` keeps local
+checkouts of them up to date for you:
+
+```env
+# .env
+BOR_GIT_SOURCES=https://git.reeseapps.com/reese/homelab.git,git@github.com:reese/deployments.git
+BOR_SOURCES_DIR=~/bor-sources   # default; each repo lands in <dir>/<repo-name>/
+```
+
+- `BOR_GIT_SOURCES` is a **comma-separated list** of URLs. Auth is whatever
+  the machine supplies — `https://…` via the OS credential helper, or
+  `git@host:repo.git` via your SSH key; no credentials are stored in the
+  app or `.env`.
+- Every run **clones** each repo (first time, shallow `--depth 1`) or
+  **pulls** it (`git pull --ff-only` — fast-forward only, so a diverged or
+  broken checkout fails loudly instead of merging) into
+  `BOR_SOURCES_DIR/<repo-name>/`, then indexes the checkouts exactly like
+  any local directory (A9 format filter, hidden-dir skip, sha256 delta).
+  `documents.source` is the repo directory name (e.g. `homelab`).
+- **`--source <path>` overrides**: when the flag is given,
+  `BOR_GIT_SOURCES` is ignored and the manual directory(ies) are imported.
+- **A failed sync aborts the run**: if any repo cannot be cloned/pulled,
+  `import_docs` exits non-zero naming the failing repo and imports
+  **nothing** (no partial junk). Fix the URL/connectivity and re-run — the
+  other checkouts stay on disk and are pulled as usual.
+
 ## Checking retrieval quality
 
 Ask the *real* pipeline (live aipi embeddings + the current KB) whether a
@@ -374,6 +423,8 @@ served locally (no CDN), `BOR_ENVIRONMENT=production`.
 | `BOR_HYBRID_LEXICAL_CANDIDATES` | `30` | FTS list width for the RRF fusion |
 | `BOR_RRF_K` | `60` | RRF damping constant (`1/(k + rank)`) |
 | `BOR_IMPORT_EXTENSIONS` | `md,markdown,txt,yaml,yml,json,py` | csv of importable formats (may only narrow the A9 set) |
+| `BOR_GIT_SOURCES` | — (empty) | csv of git repo URLs; `import_docs` clones/pulls them into `BOR_SOURCES_DIR` and indexes the checkouts (see *Git-based sources*) |
+| `BOR_SOURCES_DIR` | `~/bor-sources` | where the `BOR_GIT_SOURCES` repos are cloned/pulled (one subdirectory per repo) |
 | `BOR_STEERING_MAX_CHARS` | `8000` | char budget for the `<tuning>` (steering notes) prompt section |
 | `BOR_SUGGESTIONS` | built-in list | JSON list of onboarding chips |
 | `BOR_ADMIN_PASSWORD` | *(required)* | the single admin's password (plaintext, `.env`); app refuses to start when empty |