Experimental/brain-of-reese
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat(rag): git-based import sources — BOR_GIT_SOURCES repos cloned (first run, --depth 1) or pulled (--ff-only) into BOR_SOURCES_DIR/<repo>/ then indexed; --source still wins; a failed sync aborts before importing anything

Browse Source
This commit is contained in:
ducoterra
2026-08-25 14:23:02 -04:00
parent 589e26dbe9
commit 3d044f33a1
12 changed files with 828 additions and 16 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.agent/phases/complete/28_git_based_sources/01_env_var_and_settings.md
+37
View File
@@ -0,0 +1,37 @@
# Task 01 — Settings: `BOR_GIT_SOURCES` + `BOR_SOURCES_DIR`
**Phase:** `28_git_based_sources` · **Source:** `TODO.md:5 — "We shouldn't be hard-coding Homelab and Deployments. Instead, a list of git links should be specified."`
**Story:** `.agent/user_stories/git-sources.md`
## Objective
Add the two new settings that drive the git-based source flow: the comma-separated list of git URLs and the dedicated local clone location.
## Work
1. `app/config.py` — in the "Import scope" section (near `import_extensions`) add:
- `git_sources: str = ""` — comma-separated git repository URLs (no default; when empty, `import_docs` falls back to `--source` / the old `DEFAULT_SOURCES`). Document: "list of git repo URLs to clone/pull into `sources_dir` before indexing (phase 28); empty means no git sources".
- `sources_dir: str = "~/bor-sources"` — the dedicated local directory the repos are cloned/pulled into (phase 28). Document: "where `import_docs` clones/pulls `git_sources` repos (expanded ~ via `Path.expanduser`)".
- Add a property `git_source_list` returning the non-empty, stripped URLs (list[str]) — used by the import script.
2. `tests/unit/test_config.py` — add tests:
- `git_sources` default is `""`; `git_source_list` returns `[]` when empty.
- `git_sources` env override parses a comma-separated list (whitespace trimmed, empties dropped).
- `sources_dir` default is `~/bor-sources` (raw string, not expanded in the setting — expansion happens in the script).
3. `.env.example` — document both under an "Import scope / git sources" comment block:
```
# --- Import sources (git; phase 28) ---
# BOR_GIT_SOURCES=https://github.com/user/homelab.git,https://github.com/user/deployments.git
# BOR_SOURCES_DIR=~/bor-sources
```
## ASSUMPTIONS
- `git_sources` is empty by default (backwards-compatible: the `import_docs` script keeps its `--source` default until the owner sets `BOR_GIT_SOURCES`).
- URLs are stored raw (no parsing of scheme/auth in the setting) — parsing happens in `git_sync.py`.
- `sources_dir` is stored as a raw string; `Path.expanduser()` is applied in the script (so the setting stays env-agnostic and testable).
## Testing & Quality
- Unit: `tests/unit/test_config.py` additions.
- Coverage: the new setting is exercised by the unit test; the >90% `app/` gate is maintained.
## Completion Criteria
- [ ] `Settings().git_source_list` returns `[]` by default and the parsed list when `BOR_GIT_SOURCES` is set.
- [ ] `sources_dir` defaults to `~/bor-sources`.
- [ ] `.env.example` documents both vars.
.agent/phases/complete/28_git_based_sources/02_git_clone_pull_utility.md
+67
View File
@@ -0,0 +1,67 @@
# Task 02 — Git clone/pull utility
**Phase:** `28_git_based_sources` · **Source:** `TODO.md:5 — "import_docs should clone or pull to a dedicated repository location and then index all the specified repository code."`
**Story:** `.agent/user_stories/git-sources.md`
## Objective
Create `scripts/git_sync.py` with a single `clone_or_pull(url, dest)` function that clones a repo (shallow, first run) or fast-forwards it (subsequent runs), returning the destination path. This is the only place `git` is invoked.
## Work
1. `scripts/git_sync.py` (new) — stdlib `subprocess` only (A11: no new packages):
```python
"""Git source sync for import_docs (phase 28).
clone_or_pull(url, dest) clones ``url`` into ``dest`` (shallow, depth 1)
the first time, or fast-forwards an existing checkout with ``git pull
--ff-only`` on subsequent runs. Auth is whatever the URL/SSH config
supplies — no credentials are stored here.
"""
from __future__ import annotations
import subprocess
from pathlib import Path
class GitSyncError(RuntimeError): ...
def clone_or_pull(url: str, dest: Path | str) -> Path:
dest = Path(dest)
if not dest.exists() or not (dest / ".git").exists():
dest.parent.mkdir(parents=True, exist_ok=True)
_run(["git", "clone", "--depth", "1", url, str(dest)], cwd=dest.parent)
else:
_run(["git", "pull", "--ff-only"], cwd=dest)
return dest
def _run(argv, cwd):
try:
proc = subprocess.run(argv, cwd=cwd, capture_output=True, text=True)
except FileNotFoundError:
raise GitSyncError("git was not found on PATH — install git and retry")
if proc.returncode != 0:
raise GitSyncError(f"git {' '.join(argv[1:])} failed (exit {proc.returncode}): {proc.stderr.strip()}")
return proc.stdout
```
- `--depth 1` clone (fast, and the KB is re-imported incrementally anyway).
- `--ff-only` pull (refuses to merge unrelated histories — a broken checkout fails loudly rather than producing a dirty index).
- `GitSyncError` carries the `git` stderr so the caller can name the failing repo + reason.
- Auth: nothing special — a `https://…` URL uses the OS credential helper / prompts; an `git@host:repo.git` URL uses the machine's SSH key. Document this in the module docstring.
2. `tests/unit/test_git_sync.py` (new):
- `clone_or_pull` **clones** when the dest has no `.git` — verify by monkeypatching `subprocess.run` to a fake that records the argv and returns `returncode=0`; assert `["git","clone","--depth","1",url,str(dest)]` was called and `dest` returned.
- `clone_or_pull` **pulls** when `.git` exists — monkeypatch; assert `["git","pull","--ff-only"]` called.
- `clone_or_pull` raises `GitSyncError` on non-zero exit, carrying the stderr text.
- `clone_or_pull` raises `GitSyncError("git was not found…")` when `subprocess.run` raises `FileNotFoundError`.
- `dest.parent` is created before clone (assert the fake saw a parent that `mkdir` would create — or test the `mkdir` call path directly).
## ASSUMPTIONS
- `--depth 1` shallow clone is sufficient (the KB is re-imported incrementally; no need for full history).
- `--ff-only` pull is the right policy (refuse merges — a dirty/broken checkout fails loudly).
- Auth is delegated to the machine (SSH key / credential helper); no secrets are stored in code or `.env`.
- `git` CLI is assumed present (standard on homelab machines; a clear error is raised otherwise).
## Testing & Quality
- Unit: `tests/unit/test_git_sync.py` (clone vs pull dispatch, error propagation, missing-git error).
- Coverage: the new `scripts/git_sync.py` is fully covered.
## Completion Criteria
- [ ] `clone_or_pull` clones a fresh repo and pulls an existing one (verified via monkeypatched `subprocess`).
- [ ] A failing `git` call raises `GitSyncError` with the stderr; a missing `git` raises `GitSyncError` naming git.
- [ ] `uv run ruff check . && uv run pyright` clean.
.agent/phases/complete/28_git_based_sources/03_import_docs_script_refactor.md
+49
View File
@@ -0,0 +1,49 @@
# Task 03 — `import_docs` resolves git sources → local dirs
**Phase:** `28_git_based_sources` · **Source:** `TODO.md:5 — "import_docs should clone or pull to a dedicated repository location and then index all the specified repository code."`
**Story:** `.agent/user_stories/git-sources.md`
## Objective
Rewire `scripts/import_docs.py` so that, when `BOR_GIT_SOURCES` is set, it clones/pulls each repo into `BOR_SOURCES_DIR/<name>/` and indexes the resulting directories — while keeping `--source <path>` overriding for manual local directories.
## Work
1. `scripts/import_docs.py`:
- Import `clone_or_pull` from the sibling module: `from scripts.git_sync import clone_or_pull` (or, since `main()` runs as `python -m scripts.import_docs`, a top-level `from git_sync import clone_or_pull` works — `scripts/` is on `sys.path`).
- Keep `DEFAULT_SOURCES` as the fallback for the no-`--source` + no-`BOR_GIT_SOURCES` case (backwards-compatible).
- New resolution logic in `main()`:
```python
settings = get_settings()
sources = _resolve_sources(args.source, settings)
```
where `_resolve_sources`:
- If `args.source` is given → expanduser each and return (unchanged manual behaviour; `--source` wins).
- Else if `settings.git_source_list` is non-empty → for each URL, `clone_or_pull(url, Path(settings.sources_dir).expanduser() / repo_name(url))`; collect the dest dirs; return them. Any `GitSyncError` propagates (the script exits non-zero naming the failing repo — see below).
- Else → return the old `DEFAULT_SOURCES`.
- `repo_name(url)` — derive a directory name from the URL: strip a trailing `.git`, take the basename after the last `/` (or `:` for scp-style `git@host:repo.git`). Fall back to a slug of the URL if no basename.
- Before importing, log which dirs are being imported (so the operator sees the cloned paths). The existing "source dir not found" warning still applies if a clone left an empty dir.
2. Keep the existing `--prune` / `--limit` flags and the summary print unchanged.
3. Exit code: if any `GitSyncError` is raised, let it propagate to a top-level `except` that prints `import_docs: git sync failed: <reason>` to stderr and returns 1 **before** importing anything (so a bad repo doesn't silently import partial junk). Structure:
```python
try:
sources = _resolve_sources(...)
except GitSyncError as e:
print(f"import_docs: git sync failed: {e}", file=sys.stderr)
return 1
```
## ASSUMPTIONS
- `--source` always wins over `BOR_GIT_SOURCES` (explicit CLI flag beats env).
- When `BOR_GIT_SOURCES` is set, `--source` is ignored (only one source mode at a time) — document this.
- `BOR_SOURCES_DIR` defaults to `~/bor-sources`; each repo is a subdirectory named after the repo.
- A repo that fails to clone/pull aborts the whole run (no partial import) — the operator fixes the URL and re-runs; the already-cloned repos are left on disk and will be pulled on the next run.
- The `documents.source` column will be the repo directory name (e.g. `homelab`), matching the current `source=root.name` behaviour in the importer.
## Testing & Quality
- Integration: `tests/integration/test_import_docs_git.py` (new) — monkeypatch `clone_or_pull` to a fake that creates a temp dir with a fixture `.md` file and returns it; assert `import_docs.main(["--source"])`-equivalent resolution picks the cloned dir and that `import_sources` is called with it. Also assert a `GitSyncError` from the fake → exit code 1 and no import attempt.
- Coverage: the new `main()` resolution branch is covered.
## Completion Criteria
- [ ] `BOR_GIT_SOURCES` set → `import_docs` clones/pulls each repo into `BOR_SOURCES_DIR/<name>/` and imports them.
- [ ] `--source <path>` still imports that manual directory (unchanged).
- [ ] A failing git sync → non-zero exit, message naming the repo, no partial import.
- [ ] `uv run ruff check . && uv run pyright` clean.
.agent/phases/complete/28_git_based_sources/04_integration_test_and_docs.md
+37
View File
@@ -0,0 +1,37 @@
# Task 04 — Integration test + docs
**Phase:** `28_git_based_sources` · **Source:** `TODO.md:5 — "import_docs should clone or pull to a dedicated repository location and then index all the specified repository code."`
**Story:** `.agent/user_stories/git-sources.md`
## Objective
Add the integration test that exercises the full `import_docs` git-resolution path with a mocked `clone_or_pull`, and finalise the docs.
## Work
1. `tests/integration/test_import_docs_git.py` (new) — drive `scripts.import_docs.main(argv)` end to end with the git flow mocked:
- Monkeypatch `scripts.import_docs.clone_or_pull` (import the name into the module's namespace after importing it) so it **creates** a temp directory containing a fixture `.md` file and returns that path (simulating a clone/pull landing real content).
- Set `settings.sources_dir` to a `tmp_path`-based dir (via `monkeypatch.setattr` on the settings or by patching `get_settings`).
- Call `main([])` with `BOR_GIT_SOURCES` patched to a single URL.
- Assert: `clone_or_pull` was called once with the URL; the returned dir was passed to `import_sources`; the summary printed includes the fixture file (added > 0).
- Negative case: monkeypatch `clone_or_pull` to raise `GitSyncError`; assert `main([])` returns `1` and prints a message naming the repo, and that `import_sources` was **not** called.
- Manual override case: `main(["--source", str(tmp_path)])` imports the manual dir and does **not** call `clone_or_pull`.
- Use the same `FakeEmbedder`-style approach the importer tests use so no real LLM is needed (the importer's two-phase embed is duck-typed; pass a fake `llm` if `main` allows injection, or let `import_docs` build the real `LLMClient` but patch `import_sources` to capture its `sources` arg and short-circuit). Simpler: monkeypatch `scripts.import_docs.import_sources` to a fake that records the `sources` list and returns a trivial `ImportSummary` — this isolates the git-resolution logic from the whole embed pipeline.
2. `README.md`:
- Section 5 (Import your knowledge base): document the two source modes — (a) git sources via `BOR_GIT_SOURCES` (clone/pull into `BOR_SOURCES_DIR`), (b) `--source <path>` for manual directories. Keep the `~/Homelab + ~/Deployments` note as the *previous* default, now replaced by `BOR_GIT_SOURCES`.
- Add a short "Git-based sources" subsection: set `BOR_GIT_SOURCES` (comma-separated URLs) + `BOR_SOURCES_DIR`; `import_docs` clones (first run) or pulls (subsequent runs) each repo and indexes them; `--source` overrides; a failed sync aborts the run.
- The "Clicking a chip opens that document in a new tab" bullet (line ~69) is now stale — update to "opens in an almost-fullscreen modal" (phase 26).
3. `.env.example` — already updated in task 01; double-check the comment block is present and correct.
4. `.agent/user_stories/git-sources.md` — write the story file.
## ASSUMPTIONS
- The integration test mocks `clone_or_pull` + `import_sources` so it runs without network, without `git`, and without a real embed endpoint — it isolates the resolution logic.
- The README keeps a migration note for owners currently relying on the hardcoded `~/Homelab`/`~/Deployments` default (set `BOR_GIT_SOURCES` to the same two repos).
## Testing & Quality
- Integration: `tests/integration/test_import_docs_git.py`.
- Coverage: the new `main()` branch covered.
## Completion Criteria
- [ ] `uv run pytest tests/integration/test_import_docs_git.py -v --no-cov` green.
- [ ] `uv run pytest` green; `uv run pytest --cov=app --cov-report=term-missing` TOTAL ≥ pre-change number.
- [ ] `uv run ruff check . && uv run pyright` clean.
- [ ] README documents both source modes; the "opens in a new tab" bullet updated to "modal".