Experimental/brain-of-reese
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat(rag): git-based import sources — BOR_GIT_SOURCES repos cloned (first run, --depth 1) or pulled (--ff-only) into BOR_SOURCES_DIR/<repo>/ then indexed; --source still wins; a failed sync aborts before importing anything

Browse Source
This commit is contained in:
ducoterra
2026-08-25 14:23:02 -04:00
parent 589e26dbe9
commit 3d044f33a1
12 changed files with 828 additions and 16 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.agent/phases/complete/28_git_based_sources/01_env_var_and_settings.md
+37
View File
@@ -0,0 +1,37 @@
# Task 01 — Settings: `BOR_GIT_SOURCES` + `BOR_SOURCES_DIR`
**Phase:** `28_git_based_sources` · **Source:** `TODO.md:5 — "We shouldn't be hard-coding Homelab and Deployments. Instead, a list of git links should be specified."`
**Story:** `.agent/user_stories/git-sources.md`
## Objective
Add the two new settings that drive the git-based source flow: the comma-separated list of git URLs and the dedicated local clone location.
## Work
1. `app/config.py` — in the "Import scope" section (near `import_extensions`) add:
- `git_sources: str = ""` — comma-separated git repository URLs (no default; when empty, `import_docs` falls back to `--source` / the old `DEFAULT_SOURCES`). Document: "list of git repo URLs to clone/pull into `sources_dir` before indexing (phase 28); empty means no git sources".
- `sources_dir: str = "~/bor-sources"` — the dedicated local directory the repos are cloned/pulled into (phase 28). Document: "where `import_docs` clones/pulls `git_sources` repos (expanded ~ via `Path.expanduser`)".
- Add a property `git_source_list` returning the non-empty, stripped URLs (list[str]) — used by the import script.
2. `tests/unit/test_config.py` — add tests:
- `git_sources` default is `""`; `git_source_list` returns `[]` when empty.
- `git_sources` env override parses a comma-separated list (whitespace trimmed, empties dropped).
- `sources_dir` default is `~/bor-sources` (raw string, not expanded in the setting — expansion happens in the script).
3. `.env.example` — document both under an "Import scope / git sources" comment block:
```
# --- Import sources (git; phase 28) ---
# BOR_GIT_SOURCES=https://github.com/user/homelab.git,https://github.com/user/deployments.git
# BOR_SOURCES_DIR=~/bor-sources
```
## ASSUMPTIONS
- `git_sources` is empty by default (backwards-compatible: the `import_docs` script keeps its `--source` default until the owner sets `BOR_GIT_SOURCES`).
- URLs are stored raw (no parsing of scheme/auth in the setting) — parsing happens in `git_sync.py`.
- `sources_dir` is stored as a raw string; `Path.expanduser()` is applied in the script (so the setting stays env-agnostic and testable).
## Testing & Quality
- Unit: `tests/unit/test_config.py` additions.
- Coverage: the new setting is exercised by the unit test; the >90% `app/` gate is maintained.
## Completion Criteria
- [ ] `Settings().git_source_list` returns `[]` by default and the parsed list when `BOR_GIT_SOURCES` is set.
- [ ] `sources_dir` defaults to `~/bor-sources`.
- [ ] `.env.example` documents both vars.
.agent/phases/complete/28_git_based_sources/02_git_clone_pull_utility.md
+67
View File
@@ -0,0 +1,67 @@
# Task 02 — Git clone/pull utility
**Phase:** `28_git_based_sources` · **Source:** `TODO.md:5 — "import_docs should clone or pull to a dedicated repository location and then index all the specified repository code."`
**Story:** `.agent/user_stories/git-sources.md`
## Objective
Create `scripts/git_sync.py` with a single `clone_or_pull(url, dest)` function that clones a repo (shallow, first run) or fast-forwards it (subsequent runs), returning the destination path. This is the only place `git` is invoked.
## Work
1. `scripts/git_sync.py` (new) — stdlib `subprocess` only (A11: no new packages):
```python
"""Git source sync for import_docs (phase 28).
clone_or_pull(url, dest) clones ``url`` into ``dest`` (shallow, depth 1)
the first time, or fast-forwards an existing checkout with ``git pull
--ff-only`` on subsequent runs. Auth is whatever the URL/SSH config
supplies — no credentials are stored here.
"""
from __future__ import annotations
import subprocess
from pathlib import Path
class GitSyncError(RuntimeError): ...
def clone_or_pull(url: str, dest: Path | str) -> Path:
dest = Path(dest)
if not dest.exists() or not (dest / ".git").exists():
dest.parent.mkdir(parents=True, exist_ok=True)
_run(["git", "clone", "--depth", "1", url, str(dest)], cwd=dest.parent)
else:
_run(["git", "pull", "--ff-only"], cwd=dest)
return dest
def _run(argv, cwd):
try:
proc = subprocess.run(argv, cwd=cwd, capture_output=True, text=True)
except FileNotFoundError:
raise GitSyncError("git was not found on PATH — install git and retry")
if proc.returncode != 0:
raise GitSyncError(f"git {' '.join(argv[1:])} failed (exit {proc.returncode}): {proc.stderr.strip()}")
return proc.stdout
```
- `--depth 1` clone (fast, and the KB is re-imported incrementally anyway).
- `--ff-only` pull (refuses to merge unrelated histories — a broken checkout fails loudly rather than producing a dirty index).
- `GitSyncError` carries the `git` stderr so the caller can name the failing repo + reason.
- Auth: nothing special — a `https://…` URL uses the OS credential helper / prompts; an `git@host:repo.git` URL uses the machine's SSH key. Document this in the module docstring.
2. `tests/unit/test_git_sync.py` (new):
- `clone_or_pull` **clones** when the dest has no `.git` — verify by monkeypatching `subprocess.run` to a fake that records the argv and returns `returncode=0`; assert `["git","clone","--depth","1",url,str(dest)]` was called and `dest` returned.
- `clone_or_pull` **pulls** when `.git` exists — monkeypatch; assert `["git","pull","--ff-only"]` called.
- `clone_or_pull` raises `GitSyncError` on non-zero exit, carrying the stderr text.
- `clone_or_pull` raises `GitSyncError("git was not found…")` when `subprocess.run` raises `FileNotFoundError`.
- `dest.parent` is created before clone (assert the fake saw a parent that `mkdir` would create — or test the `mkdir` call path directly).
## ASSUMPTIONS
- `--depth 1` shallow clone is sufficient (the KB is re-imported incrementally; no need for full history).
- `--ff-only` pull is the right policy (refuse merges — a dirty/broken checkout fails loudly).
- Auth is delegated to the machine (SSH key / credential helper); no secrets are stored in code or `.env`.
- `git` CLI is assumed present (standard on homelab machines; a clear error is raised otherwise).
## Testing & Quality
- Unit: `tests/unit/test_git_sync.py` (clone vs pull dispatch, error propagation, missing-git error).
- Coverage: the new `scripts/git_sync.py` is fully covered.
## Completion Criteria
- [ ] `clone_or_pull` clones a fresh repo and pulls an existing one (verified via monkeypatched `subprocess`).
- [ ] A failing `git` call raises `GitSyncError` with the stderr; a missing `git` raises `GitSyncError` naming git.
- [ ] `uv run ruff check . && uv run pyright` clean.
.agent/phases/complete/28_git_based_sources/03_import_docs_script_refactor.md
+49
View File
@@ -0,0 +1,49 @@
# Task 03 — `import_docs` resolves git sources → local dirs
**Phase:** `28_git_based_sources` · **Source:** `TODO.md:5 — "import_docs should clone or pull to a dedicated repository location and then index all the specified repository code."`
**Story:** `.agent/user_stories/git-sources.md`
## Objective
Rewire `scripts/import_docs.py` so that, when `BOR_GIT_SOURCES` is set, it clones/pulls each repo into `BOR_SOURCES_DIR/<name>/` and indexes the resulting directories — while keeping `--source <path>` overriding for manual local directories.
## Work
1. `scripts/import_docs.py`:
- Import `clone_or_pull` from the sibling module: `from scripts.git_sync import clone_or_pull` (or, since `main()` runs as `python -m scripts.import_docs`, a top-level `from git_sync import clone_or_pull` works — `scripts/` is on `sys.path`).
- Keep `DEFAULT_SOURCES` as the fallback for the no-`--source` + no-`BOR_GIT_SOURCES` case (backwards-compatible).
- New resolution logic in `main()`:
```python
settings = get_settings()
sources = _resolve_sources(args.source, settings)
```
where `_resolve_sources`:
- If `args.source` is given → expanduser each and return (unchanged manual behaviour; `--source` wins).
- Else if `settings.git_source_list` is non-empty → for each URL, `clone_or_pull(url, Path(settings.sources_dir).expanduser() / repo_name(url))`; collect the dest dirs; return them. Any `GitSyncError` propagates (the script exits non-zero naming the failing repo — see below).
- Else → return the old `DEFAULT_SOURCES`.
- `repo_name(url)` — derive a directory name from the URL: strip a trailing `.git`, take the basename after the last `/` (or `:` for scp-style `git@host:repo.git`). Fall back to a slug of the URL if no basename.
- Before importing, log which dirs are being imported (so the operator sees the cloned paths). The existing "source dir not found" warning still applies if a clone left an empty dir.
2. Keep the existing `--prune` / `--limit` flags and the summary print unchanged.
3. Exit code: if any `GitSyncError` is raised, let it propagate to a top-level `except` that prints `import_docs: git sync failed: <reason>` to stderr and returns 1 **before** importing anything (so a bad repo doesn't silently import partial junk). Structure:
```python
try:
sources = _resolve_sources(...)
except GitSyncError as e:
print(f"import_docs: git sync failed: {e}", file=sys.stderr)
return 1
```
## ASSUMPTIONS
- `--source` always wins over `BOR_GIT_SOURCES` (explicit CLI flag beats env).
- When `BOR_GIT_SOURCES` is set, `--source` is ignored (only one source mode at a time) — document this.
- `BOR_SOURCES_DIR` defaults to `~/bor-sources`; each repo is a subdirectory named after the repo.
- A repo that fails to clone/pull aborts the whole run (no partial import) — the operator fixes the URL and re-runs; the already-cloned repos are left on disk and will be pulled on the next run.
- The `documents.source` column will be the repo directory name (e.g. `homelab`), matching the current `source=root.name` behaviour in the importer.
## Testing & Quality
- Integration: `tests/integration/test_import_docs_git.py` (new) — monkeypatch `clone_or_pull` to a fake that creates a temp dir with a fixture `.md` file and returns it; assert `import_docs.main(["--source"])`-equivalent resolution picks the cloned dir and that `import_sources` is called with it. Also assert a `GitSyncError` from the fake → exit code 1 and no import attempt.
- Coverage: the new `main()` resolution branch is covered.
## Completion Criteria
- [ ] `BOR_GIT_SOURCES` set → `import_docs` clones/pulls each repo into `BOR_SOURCES_DIR/<name>/` and imports them.
- [ ] `--source <path>` still imports that manual directory (unchanged).
- [ ] A failing git sync → non-zero exit, message naming the repo, no partial import.
- [ ] `uv run ruff check . && uv run pyright` clean.
.agent/phases/complete/28_git_based_sources/04_integration_test_and_docs.md
+37
View File
@@ -0,0 +1,37 @@
# Task 04 — Integration test + docs
**Phase:** `28_git_based_sources` · **Source:** `TODO.md:5 — "import_docs should clone or pull to a dedicated repository location and then index all the specified repository code."`
**Story:** `.agent/user_stories/git-sources.md`
## Objective
Add the integration test that exercises the full `import_docs` git-resolution path with a mocked `clone_or_pull`, and finalise the docs.
## Work
1. `tests/integration/test_import_docs_git.py` (new) — drive `scripts.import_docs.main(argv)` end to end with the git flow mocked:
- Monkeypatch `scripts.import_docs.clone_or_pull` (import the name into the module's namespace after importing it) so it **creates** a temp directory containing a fixture `.md` file and returns that path (simulating a clone/pull landing real content).
- Set `settings.sources_dir` to a `tmp_path`-based dir (via `monkeypatch.setattr` on the settings or by patching `get_settings`).
- Call `main([])` with `BOR_GIT_SOURCES` patched to a single URL.
- Assert: `clone_or_pull` was called once with the URL; the returned dir was passed to `import_sources`; the summary printed includes the fixture file (added > 0).
- Negative case: monkeypatch `clone_or_pull` to raise `GitSyncError`; assert `main([])` returns `1` and prints a message naming the repo, and that `import_sources` was **not** called.
- Manual override case: `main(["--source", str(tmp_path)])` imports the manual dir and does **not** call `clone_or_pull`.
- Use the same `FakeEmbedder`-style approach the importer tests use so no real LLM is needed (the importer's two-phase embed is duck-typed; pass a fake `llm` if `main` allows injection, or let `import_docs` build the real `LLMClient` but patch `import_sources` to capture its `sources` arg and short-circuit). Simpler: monkeypatch `scripts.import_docs.import_sources` to a fake that records the `sources` list and returns a trivial `ImportSummary` — this isolates the git-resolution logic from the whole embed pipeline.
2. `README.md`:
- Section 5 (Import your knowledge base): document the two source modes — (a) git sources via `BOR_GIT_SOURCES` (clone/pull into `BOR_SOURCES_DIR`), (b) `--source <path>` for manual directories. Keep the `~/Homelab + ~/Deployments` note as the *previous* default, now replaced by `BOR_GIT_SOURCES`.
- Add a short "Git-based sources" subsection: set `BOR_GIT_SOURCES` (comma-separated URLs) + `BOR_SOURCES_DIR`; `import_docs` clones (first run) or pulls (subsequent runs) each repo and indexes them; `--source` overrides; a failed sync aborts the run.
- The "Clicking a chip opens that document in a new tab" bullet (line ~69) is now stale — update to "opens in an almost-fullscreen modal" (phase 26).
3. `.env.example` — already updated in task 01; double-check the comment block is present and correct.
4. `.agent/user_stories/git-sources.md` — write the story file.
## ASSUMPTIONS
- The integration test mocks `clone_or_pull` + `import_sources` so it runs without network, without `git`, and without a real embed endpoint — it isolates the resolution logic.
- The README keeps a migration note for owners currently relying on the hardcoded `~/Homelab`/`~/Deployments` default (set `BOR_GIT_SOURCES` to the same two repos).
## Testing & Quality
- Integration: `tests/integration/test_import_docs_git.py`.
- Coverage: the new `main()` branch covered.
## Completion Criteria
- [ ] `uv run pytest tests/integration/test_import_docs_git.py -v --no-cov` green.
- [ ] `uv run pytest` green; `uv run pytest --cov=app --cov-report=term-missing` TOTAL ≥ pre-change number.
- [ ] `uv run ruff check . && uv run pyright` clean.
- [ ] README documents both source modes; the "opens in a new tab" bullet updated to "modal".
.env.example
+9
View File
@@ -36,6 +36,15 @@ BOR_RRF_K=60 # Reciprocal Rank Fusion damping constant
# BOR_IMPORT_EXTENSIONS=md,markdown,txt,yaml,yml,json,py # BOR_IMPORT_EXTENSIONS=md,markdown,txt,yaml,yml,json,py
# BOR_SUGGESTIONS=["How is my Kubernetes cluster set up?"] # JSON list of onboarding chips # BOR_SUGGESTIONS=["How is my Kubernetes cluster set up?"] # JSON list of onboarding chips
# --- Import sources (git; phase 28) ---
# Comma-separated git repo URLs; import_docs clones each (first run) or
# pulls it (subsequent runs) into BOR_SOURCES_DIR/<repo-name>/ and indexes
# the result. Empty = no git sources (import_docs falls back to --source /
# the old ~/Homelab + ~/Deployments defaults). Auth via URL (e.g. an
# https token) or SSH keys.
# BOR_GIT_SOURCES=https://github.com/user/homelab.git,https://github.com/user/deployments.git
# BOR_SOURCES_DIR=~/bor-sources
# --- Admin & sign-in (single-admin password login; BOTH required) --- # --- Admin & sign-in (single-admin password login; BOTH required) ---
# The app refuses to start while either is empty (names the missing # The app refuses to start while either is empty (names the missing
# variable(s) — README "Admin & sign-in"). Generate the secret with: # variable(s) — README "Admin & sign-in"). Generate the secret with:
README.md
+60 -9
View File
@@ -60,9 +60,22 @@ uv run alembic upgrade head
### 5. Import your knowledge base ### 5. Import your knowledge base
```bash ```bash
uv run python -m scripts.llm_probe # sanity: models + 768-dim check uv run python -m scripts.llm_probe # sanity: models + 768-dim check
uv run python -m scripts.import_docs # defaults: ~/Homelab + ~/Deployments uv run python -m scripts.import_docs # import the configured sources (below)
``` ```
Two source modes:
- **Git sources (recommended)** — set `BOR_GIT_SOURCES` in `.env` to a
comma-separated list of git repo URLs. `import_docs` clones each repo
(first run) or pulls it (subsequent runs) into
`BOR_SOURCES_DIR/<repo-name>/` (default `~/bor-sources`) and indexes the
checkouts — see [Git-based sources](#git-based-sources).
- **Manual directories** — `--source <path>` (repeatable) imports local
directories directly and *always wins* over `BOR_GIT_SOURCES`.
- If neither is set, the import falls back to the **previous** default,
`~/Homelab` + `~/Deployments` — kept only for backwards compatibility,
now replaced by `BOR_GIT_SOURCES`.
### 6. Run the app ### 6. Run the app
```bash ```bash
uv run uvicorn app.main:app --reload uv run uvicorn app.main:app --reload
@@ -76,16 +89,19 @@ uv run uvicorn app.main:app --reload
- **Chat** (`/`) — ask questions; answers stream in with **source chips** - **Chat** (`/`) — ask questions; answers stream in with **source chips**
that cite the exact documents used. Clicking a chip opens that document that cite the exact documents used. Clicking a chip opens that document
**in a new tab**. in an **almost-fullscreen modal** on the same page (no new tab).
- **Document viewer** (`/document.html?source=…&path=…`) — the full text of - **Document viewer** — the modal above *is* the viewer; the full text of
any indexed document, served from the database (no filesystem access): any indexed document is served from the database (no filesystem access):
markdown is rendered, every other format (`yaml`, `json`, `py`, `txt`, …) markdown is rendered, every other format (`yaml`, `json`, `py`, `txt`, …)
is shown as escaped monospace text. Unknown documents get a designed is shown as escaped monospace text. `/document.html?source=…&path=…`
not-found state with a link back to the index. stays as the **full-page / direct-link** form (the modal's “Full page”
button and the URL to share — it works without JS). Unknown documents
get a designed not-found state with a link back to the index.
- **Sources** (`/sources.html`) — the indexed document list; the *Path* - **Sources** (`/sources.html`) — the indexed document list; the *Path*
column links each document to the viewer in a new tab. **Admin-only** — column opens each document in the same **almost-fullscreen modal** (no
anonymous visitors see a sign-in gate instead (the catalog is what the new tab). **Admin-only** — anonymous visitors see a sign-in gate instead
login locks; the document viewer itself stays open to everyone). (the catalog is what the login locks; the document viewer itself stays
open to everyone).
## Thinking ## Thinking
@@ -205,6 +221,10 @@ uv run python -m scripts.import_docs --prune # also drop deleted/out-of-
uv run python -m scripts.import_docs --source ~/SomeOtherDocs uv run python -m scripts.import_docs --source ~/SomeOtherDocs
``` ```
With **git-based sources** (below) each run first pulls the latest commits
of your repos, so this same command is the whole update loop: commit in the
repo → re-run the import.
Then check the **Sources** page (`http://localhost:8000/sources.html`): Then check the **Sources** page (`http://localhost:8000/sources.html`):
the *documents* / *chunks* counters and *last indexed* timestamp should the *documents* / *chunks* counters and *last indexed* timestamp should
reflect the new files, and each document row shows when it was last reflect the new files, and each document row shows when it was last
@@ -230,6 +250,35 @@ embedded.
- To sanity-check the LLM backend (models + embedding dimension) after any - To sanity-check the LLM backend (models + embedding dimension) after any
aipi change: `uv run python -m scripts.llm_probe`. aipi change: `uv run python -m scripts.llm_probe`.
### Git-based sources
Rather than pointing the import at local folders, point it at **git
repositories** — the notes live in the repos and `import_docs` keeps local
checkouts of them up to date for you:
```env
# .env
BOR_GIT_SOURCES=https://git.reeseapps.com/reese/homelab.git,git@github.com:reese/deployments.git
BOR_SOURCES_DIR=~/bor-sources # default; each repo lands in <dir>/<repo-name>/
```
- `BOR_GIT_SOURCES` is a **comma-separated list** of URLs. Auth is whatever
the machine supplies — `https://…` via the OS credential helper, or
`git@host:repo.git` via your SSH key; no credentials are stored in the
app or `.env`.
- Every run **clones** each repo (first time, shallow `--depth 1`) or
**pulls** it (`git pull --ff-only` — fast-forward only, so a diverged or
broken checkout fails loudly instead of merging) into
`BOR_SOURCES_DIR/<repo-name>/`, then indexes the checkouts exactly like
any local directory (A9 format filter, hidden-dir skip, sha256 delta).
`documents.source` is the repo directory name (e.g. `homelab`).
- **`--source <path>` overrides**: when the flag is given,
`BOR_GIT_SOURCES` is ignored and the manual directory(ies) are imported.
- **A failed sync aborts the run**: if any repo cannot be cloned/pulled,
`import_docs` exits non-zero naming the failing repo and imports
**nothing** (no partial junk). Fix the URL/connectivity and re-run — the
other checkouts stay on disk and are pulled as usual.
## Checking retrieval quality ## Checking retrieval quality
Ask the *real* pipeline (live aipi embeddings + the current KB) whether a Ask the *real* pipeline (live aipi embeddings + the current KB) whether a
@@ -374,6 +423,8 @@ served locally (no CDN), `BOR_ENVIRONMENT=production`.
| `BOR_HYBRID_LEXICAL_CANDIDATES` | `30` | FTS list width for the RRF fusion | | `BOR_HYBRID_LEXICAL_CANDIDATES` | `30` | FTS list width for the RRF fusion |
| `BOR_RRF_K` | `60` | RRF damping constant (`1/(k + rank)`) | | `BOR_RRF_K` | `60` | RRF damping constant (`1/(k + rank)`) |
| `BOR_IMPORT_EXTENSIONS` | `md,markdown,txt,yaml,yml,json,py` | csv of importable formats (may only narrow the A9 set) | | `BOR_IMPORT_EXTENSIONS` | `md,markdown,txt,yaml,yml,json,py` | csv of importable formats (may only narrow the A9 set) |
| `BOR_GIT_SOURCES` | — (empty) | csv of git repo URLs; `import_docs` clones/pulls them into `BOR_SOURCES_DIR` and indexes the checkouts (see *Git-based sources*) |
| `BOR_SOURCES_DIR` | `~/bor-sources` | where the `BOR_GIT_SOURCES` repos are cloned/pulled (one subdirectory per repo) |
| `BOR_STEERING_MAX_CHARS` | `8000` | char budget for the `<tuning>` (steering notes) prompt section | | `BOR_STEERING_MAX_CHARS` | `8000` | char budget for the `<tuning>` (steering notes) prompt section |
| `BOR_SUGGESTIONS` | built-in list | JSON list of onboarding chips | | `BOR_SUGGESTIONS` | built-in list | JSON list of onboarding chips |
| `BOR_ADMIN_PASSWORD` | *(required)* | the single admin's password (plaintext, `.env`); app refuses to start when empty | | `BOR_ADMIN_PASSWORD` | *(required)* | the single admin's password (plaintext, `.env`); app refuses to start when empty |
app/config.py
+19
View File
@@ -105,6 +105,15 @@ class Settings(BaseSettings):
# via :py:meth:`import_extension_set`. ``mode="after"`` validation runs # via :py:meth:`import_extension_set`. ``mode="after"`` validation runs
# against the raw string so a typo fails loudly at startup. # against the raw string so a typo fails loudly at startup.
import_extensions: str = "md,markdown,txt,yaml,yml,json,py" import_extensions: str = "md,markdown,txt,yaml,yml,json,py"
#: List of git repo URLs to clone/pull into ``sources_dir`` before
#: indexing (phase 28); comma-separated, stored raw. Empty means no git
#: sources — ``import_docs`` then falls back to ``--source`` / the old
#: ``DEFAULT_SOURCES``.
git_sources: str = ""
#: Where ``import_docs`` clones/pulls the ``git_sources`` repos (phase
#: 28). Stored as a raw string — ``Path.expanduser()`` is applied in
#: the import script, not here.
sources_dir: str = "~/bor-sources"
@field_validator("import_extensions") @field_validator("import_extensions")
@classmethod @classmethod
@@ -139,6 +148,16 @@ class Settings(BaseSettings):
if part.strip() if part.strip()
) )
@property
def git_source_list(self) -> list[str]:
"""Non-empty, stripped git URLs from :py:attr:`git_sources` (phase 28).
Whitespace around each entry is trimmed and empty entries dropped;
an unset/empty value yields ``[]`` (the import script then uses its
legacy local-directory defaults).
"""
return [part.strip() for part in self.git_sources.split(",") if part.strip()]
@property @property
def effective_api_key(self) -> str: def effective_api_key(self) -> str:
"""API key for aipi: explicit setting, then $AIPI_KEY, then a placeholder.""" """API key for aipi: explicit setting, then $AIPI_KEY, then a placeholder."""
scripts/git_sync.py
+60
View File
@@ -0,0 +1,60 @@
"""Git source sync for import_docs (phase 28).
clone_or_pull(url, dest) clones ``url`` into ``dest`` (shallow, depth 1)
the first time, or fast-forwards an existing checkout with ``git pull
--ff-only`` on subsequent runs.
Auth: nothing special — an ``https://…`` URL uses the OS credential
helper / prompts; a ``git@host:repo.git`` URL uses the machine's SSH key.
No credentials are stored here; whatever the URL/SSH config supplies is
used.
This module is the only place the ``git`` CLI is invoked (A11: stdlib
``subprocess`` only, no new packages).
"""
from __future__ import annotations
import subprocess
from pathlib import Path
__all__ = ["GitSyncError", "clone_or_pull"]
class GitSyncError(RuntimeError):
"""A git clone/pull failed (or git is missing); carries git's stderr."""
def clone_or_pull(url: str, dest: Path | str) -> Path:
"""Clone ``url`` into ``dest`` (shallow, first run) or fast-forward it.
- dest without a ``.git`` (or absent) → ``git clone --depth 1 url dest``
(shallow: the KB is re-imported incrementally anyway).
- dest with a ``.git`` → ``git pull --ff-only`` (refuses to merge
unrelated histories — a broken checkout fails loudly rather than
producing a dirty index).
Returns the destination path. Raises :class:`GitSyncError` when git is
missing or a git invocation exits non-zero (with git's stderr in the
message, so the caller can name the failing repo + reason).
"""
dest = Path(dest)
if not dest.exists() or not (dest / ".git").exists():
dest.parent.mkdir(parents=True, exist_ok=True)
_run(["git", "clone", "--depth", "1", url, str(dest)], cwd=dest.parent)
else:
_run(["git", "pull", "--ff-only"], cwd=dest)
return dest
def _run(argv: list[str], cwd: Path) -> str:
"""Run a git command, capturing output; raise GitSyncError on failure."""
try:
proc = subprocess.run(argv, cwd=cwd, capture_output=True, text=True)
except FileNotFoundError:
raise GitSyncError("git was not found on PATH — install git and retry") from None
if proc.returncode != 0:
raise GitSyncError(
f"git {' '.join(argv[1:])} failed (exit {proc.returncode}): "
f"{proc.stderr.strip()}"
)
return proc.stdout
scripts/import_docs.py
+78 -7
View File
@@ -2,16 +2,28 @@
Examples:: Examples::
uv run python -m scripts.import_docs # ~/Homelab + ~/Deployments uv run python -m scripts.import_docs # BOR_GIT_SOURCES, else ~/Homelab + ~/Deployments
uv run python -m scripts.import_docs --source ~/OtherDocs # extra dir (repeatable) uv run python -m scripts.import_docs --source ~/OtherDocs # explicit dir(s); always wins
uv run python -m scripts.import_docs --prune # also drop deleted/out-of-scope files uv run python -m scripts.import_docs --prune # also drop deleted/out-of-scope files
uv run python -m scripts.import_docs --limit 5 # debug: first 5 files only uv run python -m scripts.import_docs --limit 5 # debug: first 5 files only
Source resolution (phase 28), in precedence order:
1. ``--source PATH`` — explicit manual directories (repeatable) always win;
``BOR_GIT_SOURCES`` is ignored when this flag is used.
2. ``BOR_GIT_SOURCES`` (comma-separated git URLs) — each repo is cloned
(first run, shallow ``--depth 1``) or fast-forwarded (``git pull
--ff-only``) into ``BOR_SOURCES_DIR/<repo-name>/`` (default
``~/bor-sources``) and the resulting checkouts are imported. A failing
clone/pull aborts the whole run *before* anything is imported.
3. Fallback — the legacy ``DEFAULT_SOURCES`` (``~/Homelab`` +
``~/Deployments``), kept for backwards compatibility.
Imported formats (PLAN anchor A9, revised): ``md, markdown, txt, yaml, Imported formats (PLAN anchor A9, revised): ``md, markdown, txt, yaml,
yml, json, py`` (case-insensitive; narrow with ``BOR_IMPORT_EXTENSIONS``). yml, json, py`` (case-insensitive; narrow with ``BOR_IMPORT_EXTENSIONS``).
Any path with a dot-prefixed component (hidden files/dirs — vendored Any path with a dot-prefixed component (hidden files/dirs — vendored
caches) is skipped, along with non-content dirs (``.venv``, caches) is skipped, along with non-content dirs (``.venv``,
``node_modules``, ``.git``, ``__pycache__``, ``.pytest_cache``, ``dist``, ``node_modules``, ``.git``, ``__pycache``, ``.pytest_cache``, ``dist``,
``build``). Re-runs are cheap: files are diffed by sha256 and unchanged ``build``). Re-runs are cheap: files are diffed by sha256 and unchanged
ones are not re-embedded; ``--prune`` also drops documents whose files no ones are not re-embedded; ``--prune`` also drops documents whose files no
longer match the format filter. longer match the format filter.
@@ -20,14 +32,19 @@ from __future__ import annotations
import argparse import argparse
import asyncio import asyncio
import logging
import re
import sys import sys
from pathlib import Path from pathlib import Path
from app.config import get_settings from app.config import Settings, get_settings
from app.core.debugging import configure_debugging from app.core.debugging import configure_debugging
from app.core.logging import configure_logging from app.core.logging import configure_logging
from app.rag.importer import import_sources from app.rag.importer import import_sources
from app.rag.llm import LLMClient from app.rag.llm import LLMClient
from scripts.git_sync import GitSyncError, clone_or_pull
logger = logging.getLogger("scripts.import_docs")
DEFAULT_SOURCES: list[Path] = [Path("~/Homelab"), Path("~/Deployments")] DEFAULT_SOURCES: list[Path] = [Path("~/Homelab"), Path("~/Deployments")]
@@ -42,7 +59,10 @@ def build_parser() -> argparse.ArgumentParser:
action="append", action="append",
type=Path, type=Path,
metavar="PATH", metavar="PATH",
help="directory to import (repeatable; default: ~/Homelab ~/Deployments)", help=(
"directory to import (repeatable; always wins over BOR_GIT_SOURCES; "
"default when neither is given: ~/Homelab ~/Deployments)"
),
) )
p.add_argument( p.add_argument(
"--prune", "--prune",
@@ -59,12 +79,63 @@ def build_parser() -> argparse.ArgumentParser:
return p return p
def repo_name(url: str) -> str:
"""Local directory name for a git URL (phase 28).
Strips a trailing ``.git`` and takes the basename after the last ``/``
(``:`` for scp-style ``git@host:repo.git`` URLs); falls back to a slug
of the whole URL when no usable basename remains.
"""
name = url.strip()
if name.endswith(".git"):
name = name[: -len(".git")]
base = name.rsplit("/", 1)[-1].rsplit(":", 1)[-1].strip()
if base:
return base
slug = re.sub(r"[^A-Za-z0-9]+", "-", name).strip("-")
return slug or "repo"
def _resolve_sources(cli_sources: list[Path] | None, settings: Settings) -> list[Path]:
"""Resolve the directories to import (phase 28).
Precedence: ``--source`` (explicit manual paths — always wins) >
``BOR_GIT_SOURCES`` (each URL cloned/pulled via
:func:`scripts.git_sync.clone_or_pull` into
``BOR_SOURCES_DIR/<repo-name>/``) > the legacy ``DEFAULT_SOURCES``.
A :class:`GitSyncError` from a failing clone/pull propagates to
:func:`main`, which aborts the run before importing anything.
"""
if cli_sources:
return [path.expanduser() for path in cli_sources]
git_urls = settings.git_source_list
if git_urls:
sources_root = Path(settings.sources_dir).expanduser()
return [clone_or_pull(url, sources_root / repo_name(url)) for url in git_urls]
return [path.expanduser() for path in DEFAULT_SOURCES]
def main(argv: list[str] | None = None) -> int: def main(argv: list[str] | None = None) -> int:
args = build_parser().parse_args(argv) args = build_parser().parse_args(argv)
configure_logging(get_settings().log_level) settings = get_settings()
configure_logging(settings.log_level)
configure_debugging() configure_debugging()
sources = [path.expanduser() for path in (args.source or DEFAULT_SOURCES)] # Git sources resolve (and clone/pull) *before* any import: a failing
# repo aborts the run with a non-zero exit, naming the failure — a bad
# URL must never silently import partial junk.
try:
sources = _resolve_sources(args.source, settings)
except GitSyncError as e:
print(f"import_docs: git sync failed: {e}", file=sys.stderr)
return 1
logger.info(
"import_docs: importing %d source dir(s): %s",
len(sources),
", ".join(str(s) for s in sources),
)
missing = [s for s in sources if not s.is_dir()] missing = [s for s in sources if not s.is_dir()]
for s in missing: for s in missing:
print(f"import_docs: source dir not found: {s}", file=sys.stderr) print(f"import_docs: source dir not found: {s}", file=sys.stderr)
tests/integration/test_import_docs_git.py
+212
View File
@@ -0,0 +1,212 @@
"""Integration test: ``import_docs`` git-source resolution (phase 28, task 03).
Drives ``scripts.import_docs`` end to end with a fake ``clone_or_pull`` (no
real git, no network) and a recording fake ``import_sources`` (no real
DB), covering:
- ``BOR_GIT_SOURCES`` set → each URL is cloned/pulled into
``BOR_SOURCES_DIR/<repo-name>/`` and exactly those dirs are imported.
- ``--source`` still wins over ``BOR_GIT_SOURCES`` (no git at all).
- No git sources + no ``--source`` → the legacy ``DEFAULT_SOURCES``.
- A failing git sync → exit code 1, an error naming the failing repo on
stderr, and **zero** import attempts.
"""
from __future__ import annotations
from pathlib import Path
import pytest
from app.config import Settings
from app.rag.importer import ImportSummary
from scripts import import_docs
from scripts.git_sync import GitSyncError
def _settings(git_sources: str = "", sources_dir: str = "~/bor-sources") -> Settings:
"""Fresh settings (no .env file); explicit kwargs beat any env leaks."""
return Settings(_env_file=None, git_sources=git_sources, sources_dir=sources_dir) # pyright: ignore[reportCallIssue]
class FakeImportSources:
"""Records every ``import_sources`` call instead of touching a DB."""
def __init__(self) -> None:
self.calls: list[dict] = []
async def __call__(
self,
sources: list[Path],
llm: object,
*,
prune: bool = False,
limit: int | None = None,
) -> ImportSummary:
self.calls.append({"sources": list(sources), "prune": prune, "limit": limit})
return ImportSummary(files=1, added=1)
def _fake_clone_factory() -> tuple[list[tuple[str, Path]], object]:
"""A ``clone_or_pull`` that materialises a checkout with one .md file."""
calls: list[tuple[str, Path]] = []
def fake_clone_or_pull(url: str, dest: Path | str) -> Path:
dest = Path(dest)
dest.mkdir(parents=True, exist_ok=True)
(dest / "notes.md").write_text(f"# {dest.name}\ncontent for the KB\n", encoding="utf-8")
calls.append((url, dest))
return dest
return calls, fake_clone_or_pull
# --- repo_name -------------------------------------------------------------
@pytest.mark.parametrize(
("url", "name"),
[
("https://github.com/user/homelab.git", "homelab"),
("https://github.com/user/homelab", "homelab"),
("git@github.com:user/homelab.git", "homelab"),
("git@github.com:homelab.git", "homelab"),
("ssh://git@host:2222/group/deployments.git", "deployments"),
("https://git.reeseapps.com:8443/proj/notes", "notes"),
],
)
def test_repo_name(url: str, name: str) -> None:
assert import_docs.repo_name(url) == name
def test_repo_name_slug_fallback() -> None:
# No usable basename (path ends in the .git suffix itself) → slug.
assert import_docs.repo_name("https://host/.git") == "https-host"
# --- _resolve_sources ------------------------------------------------------
def test_resolve_sources_git_urls_cloned_into_sources_dir(
monkeypatch: pytest.MonkeyPatch, tmp_path: Path
) -> None:
calls, fake = _fake_clone_factory()
monkeypatch.setattr(import_docs, "clone_or_pull", fake)
settings = _settings(
git_sources="https://host/a/homelab.git, git@host:user/deploy.git ,",
sources_dir=str(tmp_path / "bor"),
)
sources = import_docs._resolve_sources(None, settings)
assert sources == [tmp_path / "bor" / "homelab", tmp_path / "bor" / "deploy"]
assert calls == [
("https://host/a/homelab.git", tmp_path / "bor" / "homelab"),
("git@host:user/deploy.git", tmp_path / "bor" / "deploy"),
]
def test_resolve_sources_cli_source_wins(
monkeypatch: pytest.MonkeyPatch, tmp_path: Path
) -> None:
calls, fake = _fake_clone_factory()
monkeypatch.setattr(import_docs, "clone_or_pull", fake)
settings = _settings(git_sources="https://host/a/repo.git")
manual = tmp_path / "Manual"
sources = import_docs._resolve_sources([manual], settings)
assert sources == [manual]
assert calls == [] # git is never touched when --source is given
def test_resolve_sources_defaults_when_nothing_configured() -> None:
sources = import_docs._resolve_sources(None, _settings())
assert sources == [p.expanduser() for p in import_docs.DEFAULT_SOURCES]
# --- main() ----------------------------------------------------------------
def test_main_git_sources_clone_then_import(
monkeypatch: pytest.MonkeyPatch,
tmp_path: Path,
capsys: pytest.CaptureFixture[str],
) -> None:
settings = _settings(
git_sources="https://host/a/homelab.git,https://host/a/deploy.git",
sources_dir=str(tmp_path / "bor"),
)
monkeypatch.setattr(import_docs, "get_settings", lambda: settings)
calls, fake = _fake_clone_factory()
monkeypatch.setattr(import_docs, "clone_or_pull", fake)
fake_import = FakeImportSources()
monkeypatch.setattr(import_docs, "import_sources", fake_import)
rc = import_docs.main([])
assert rc == 0
assert [(url, dest) for url, dest in calls] == [
("https://host/a/homelab.git", tmp_path / "bor" / "homelab"),
("https://host/a/deploy.git", tmp_path / "bor" / "deploy"),
]
assert len(fake_import.calls) == 1
# The cloned checkouts are exactly what gets imported.
assert fake_import.calls[0]["sources"] == [
tmp_path / "bor" / "homelab",
tmp_path / "bor" / "deploy",
]
for dest in (tmp_path / "bor" / "homelab", tmp_path / "bor" / "deploy"):
assert (dest / "notes.md").is_file()
# The final summary print reflects the import (added > 0).
assert "added=1" in capsys.readouterr().out
def test_main_cli_source_still_imports_manual_dir(
monkeypatch: pytest.MonkeyPatch, tmp_path: Path
) -> None:
manual = tmp_path / "manual"
manual.mkdir()
(manual / "a.md").write_text("# A\nhi\n", encoding="utf-8")
# BOR_GIT_SOURCES is set but must be ignored — --source always wins.
settings = _settings(git_sources="https://host/a/repo.git")
monkeypatch.setattr(import_docs, "get_settings", lambda: settings)
calls, fake = _fake_clone_factory()
monkeypatch.setattr(import_docs, "clone_or_pull", fake)
fake_import = FakeImportSources()
monkeypatch.setattr(import_docs, "import_sources", fake_import)
rc = import_docs.main(["--source", str(manual)])
assert rc == 0
assert calls == []
assert fake_import.calls[0]["sources"] == [manual]
assert fake_import.calls[0]["prune"] is False
def test_main_git_failure_aborts_before_import(
monkeypatch: pytest.MonkeyPatch, tmp_path: Path, capsys: pytest.CaptureFixture[str]
) -> None:
settings = _settings(
git_sources="https://host/a/bad.git",
sources_dir=str(tmp_path / "bor"),
)
monkeypatch.setattr(import_docs, "get_settings", lambda: settings)
def failing_clone(url: str, dest: Path | str) -> Path:
raise GitSyncError(
f"git clone --depth 1 {url} failed (exit 128): "
"fatal: repository not found"
)
monkeypatch.setattr(import_docs, "clone_or_pull", failing_clone)
fake_import = FakeImportSources()
monkeypatch.setattr(import_docs, "import_sources", fake_import)
rc = import_docs.main([])
assert rc == 1
err = capsys.readouterr().err
assert "import_docs: git sync failed" in err
assert "bad.git" in err # the failing repo is named
assert fake_import.calls == [] # no partial import
assert not (tmp_path / "bor").exists()
tests/unit/test_config.py
+47
View File
@@ -92,6 +92,53 @@ def test_import_extensions_rejects_empty(monkeypatch) -> None:
_settings() _settings()
def test_git_sources_default_empty_and_sources_dir_default() -> None:
"""Phase 28: no git sources by default (backwards-compatible with the
``--source`` / ``DEFAULT_SOURCES`` fallback); the clone location stays
a raw string (``~`` is expanded by the import script, not the setting)."""
s = _settings()
assert s.git_sources == ""
assert s.git_source_list == []
assert s.sources_dir == "~/bor-sources"
def test_git_sources_env_override_parses_comma_separated_list(
monkeypatch: pytest.MonkeyPatch,
) -> None:
"""``BOR_GIT_SOURCES`` is a raw CSV: entries are trimmed and empty
entries dropped; URLs are stored untouched (no scheme parsing here)."""
monkeypatch.setenv(
"BOR_GIT_SOURCES",
"https://github.com/user/homelab.git, "
" git@github.com:user/deployments.git ,, https://git.reeseapps.com/x/y.git ",
)
s = _settings()
# The raw CSV string is preserved untouched (no parsing in the setting).
assert s.git_sources == (
"https://github.com/user/homelab.git, "
" git@github.com:user/deployments.git ,, https://git.reeseapps.com/x/y.git "
)
assert s.git_source_list == [
"https://github.com/user/homelab.git",
"git@github.com:user/deployments.git",
"https://git.reeseapps.com/x/y.git",
]
def test_git_sources_whitespace_only_yields_empty_list(monkeypatch) -> None:
"""A configured-but-blank value behaves the same as unset: no git
sources, so the script falls back to its legacy local defaults."""
monkeypatch.setenv("BOR_GIT_SOURCES", " , , ")
s = _settings()
assert s.git_source_list == []
def test_sources_dir_env_override_is_raw_string(monkeypatch: pytest.MonkeyPatch) -> None:
monkeypatch.setenv("BOR_SOURCES_DIR", "/data/bor/sources")
s = _settings()
assert s.sources_dir == "/data/bor/sources"
def test_suggestions_default_is_three_plus_real_questions() -> None: def test_suggestions_default_is_three_plus_real_questions() -> None:
s = _settings() s = _settings()
assert len(s.suggestions) >= 3 assert len(s.suggestions) >= 3
tests/unit/test_git_sync.py
+153
View File
@@ -0,0 +1,153 @@
"""Unit tests: git clone/pull utility (phase 28).
``clone_or_pull`` dispatches to ``git clone --depth 1`` (fresh dest) or
``git pull --ff-only`` (existing checkout) with ``subprocess`` fully
mocked — the real git CLI is never invoked, so the tests run anywhere.
"""
from __future__ import annotations
import subprocess
from pathlib import Path
import pytest
import scripts.git_sync as git_sync
from scripts.git_sync import GitSyncError, clone_or_pull
class _FakeProc:
def __init__(self, returncode: int = 0, stdout: str = "", stderr: str = "") -> None:
self.returncode = returncode
self.stdout = stdout
self.stderr = stderr
def _fake_run(
monkeypatch: pytest.MonkeyPatch,
returncode: int = 0,
stdout: str = "",
stderr: str = "",
missing_git: bool = False,
) -> list[dict[str, object]]:
"""Monkeypatch scripts.git_sync.subprocess.run; record each call."""
calls: list[dict[str, object]] = []
def fake_run(argv: list[str], cwd: Path | None = None, **kwargs: object) -> _FakeProc:
calls.append({"argv": list(argv), "cwd": cwd, **kwargs})
if missing_git:
raise FileNotFoundError("git")
return _FakeProc(returncode, stdout, stderr)
monkeypatch.setattr(subprocess, "run", fake_run)
return calls
def test_clone_or_pull_clones_when_dest_has_no_git_dir(
tmp_path: Path, monkeypatch: pytest.MonkeyPatch
) -> None:
"""First run: dest absent → ``git clone --depth 1`` from the parent dir."""
url = "https://github.com/user/homelab.git"
dest = tmp_path / "homelab"
calls = _fake_run(monkeypatch)
result = clone_or_pull(url, dest)
assert result == dest
assert len(calls) == 1
call = calls[0]
assert call["argv"] == ["git", "clone", "--depth", "1", url, str(dest)]
assert call["cwd"] == dest.parent
assert call["capture_output"] is True
assert call["text"] is True
def test_clone_or_pull_pulls_when_git_dir_exists(
tmp_path: Path, monkeypatch: pytest.MonkeyPatch
) -> None:
"""Subsequent run: dest/.git present → ``git pull --ff-only`` in-place."""
dest = tmp_path / "homelab"
(dest / ".git").mkdir(parents=True)
calls = _fake_run(monkeypatch)
result = clone_or_pull("https://github.com/user/homelab.git", dest)
assert result == dest
assert calls[0]["argv"] == ["git", "pull", "--ff-only"]
assert calls[0]["cwd"] == dest
def test_clone_or_pull_creates_missing_parent_before_clone(
tmp_path: Path, monkeypatch: pytest.MonkeyPatch
) -> None:
"""The (nested) parent of the destination is mkdir'd before git runs."""
dest = tmp_path / "nested" / "deeper" / "homelab"
assert not dest.parent.exists()
_fake_run(monkeypatch)
clone_or_pull("https://github.com/user/homelab.git", dest)
assert dest.parent.is_dir()
def test_existing_dir_without_git_dir_is_treated_as_fresh(
tmp_path: Path, monkeypatch: pytest.MonkeyPatch
) -> None:
"""A leftover non-repo dir at the dest still takes the clone path —
git fails loudly (non-zero exit) if the dir is not empty."""
dest = tmp_path / "homelab"
dest.mkdir()
calls = _fake_run(monkeypatch)
clone_or_pull("https://github.com/user/homelab.git", dest)
assert calls[0]["argv"] == [
"git",
"clone",
"--depth",
"1",
"https://github.com/user/homelab.git",
str(dest),
]
def test_failing_git_raises_error_carrying_stderr(
tmp_path: Path, monkeypatch: pytest.MonkeyPatch
) -> None:
"""Non-zero exit → GitSyncError naming the subcommand, exit code, stderr."""
dest = tmp_path / "homelab"
_fake_run(monkeypatch, returncode=128, stderr="fatal: repository not found\n")
with pytest.raises(
GitSyncError,
match=r"git clone --depth 1 .* failed \(exit 128\): fatal: repository not found",
):
clone_or_pull("https://example.com/nope.git", dest)
# Same for the pull path (broken checkout, e.g. diverged history).
(dest / ".git").mkdir(parents=True)
_fake_run(monkeypatch, returncode=1, stderr="error: cannot pull with rebase")
with pytest.raises(
GitSyncError, match=r"git pull --ff-only failed \(exit 1\): error: cannot pull with rebase"
):
clone_or_pull("https://example.com/homelab.git", dest)
def test_missing_git_raises_named_error(
tmp_path: Path, monkeypatch: pytest.MonkeyPatch
) -> None:
"""No git binary on PATH → GitSyncError telling the user to install it."""
_fake_run(monkeypatch, missing_git=True)
with pytest.raises(GitSyncError, match="git was not found on PATH"):
clone_or_pull("https://example.com/homelab.git", tmp_path / "homelab")
def test_run_captures_and_returns_stdout(
tmp_path: Path, monkeypatch: pytest.MonkeyPatch
) -> None:
"""_run returns the captured stdout on success (git output is not lost)."""
calls = _fake_run(monkeypatch, stdout="From example.com\n + abc..def main")
assert git_sync._run(["git", "status"], cwd=tmp_path) == "From example.com\n + abc..def main"
assert len(calls) == 1