brain-of-reese/.agent/phases/complete/28_git_based_sources/00_phase.md

Phase 28 — Git-Based Sources

Source: TODO.md L5 — "We shouldn't be hard-coding Homelab and Deployments. Instead, a list of git links should be specified. import_docs should clone or pull to a dedicated repository location and then index all the specified repository code." Story: .agent/user_stories/git-sources.md Context: Phase 11 / the README document the import workflow. scripts/import_docs.py currently defaults to the hardcoded DEFAULT_SOURCES = [Path("~/Homelab"), Path("~/Deployments")] (repeatable via --source). The importer (app/rag/importer.py::import_sources) already walks any list of local directories — it needs no change to the walking logic, only to receive the resolved local paths.

Objective

Replace the hardcoded ~/Homelab + ~/Deployments default with a list of git repository URLs (BOR_GIT_SOURCES). import_docs clones (first run) or pulls (subsequent runs) each repository into a dedicated local location (BOR_SOURCES_DIR, default ~/bor-sources) and indexes the resulting directories. The existing --source flag still overrides for manual paths.

Dependencies

• 01_infrastructure (complete) — the base app + importer.
• 11_long_answers and the README import workflow (complete) — the import_sources contract and the documented CLI.

Tasks

1. 01_env_var_and_settings.md — add BOR_GIT_SOURCES (comma-separated git URLs) + BOR_SOURCES_DIR settings; update .env.example + README.
2. 02_git_clone_pull_utility.md — create scripts/git_sync.py with clone_or_pull(url, dest) (clone --depth 1 / pull --ff-only, auth via URL or SSH).
3. 03_import_docs_script_refactor.md — refactor scripts/import_docs.py to resolve git sources → local dirs, then pass to import_sources.
4. 04_integration_test_and_docs.md — integration test for the git flow (mocked git); README + .env.example finalised.

Testing & Quality

• Unit/integration: scripts/git_sync.py clone_or_pull (clone when missing, pull when present, error propagation) — unit test with a mocked subprocess; import_docs resolves git URLs → local dirs (integration).
• Coverage: >90% on app/ (the importer is unchanged; the new logic is in scripts/, covered by its own tests).
• E2E: none required (no app/ or UI change) — but the smoke suite must stay green.

Completion Criteria

• With BOR_GIT_SOURCES set and BOR_SOURCES_DIR writable, uv run python -m scripts.import_docs clones (first run) or pulls (subsequent runs) each repo into BOR_SOURCES_DIR/<name>/ and indexes them.
• --source <path> still overrides to import an arbitrary local directory (unchanged behaviour).
• Unreachable/invalid git URL → the script fails loudly (non-zero exit) naming the repo, without importing partial junk.
• .env.example documents BOR_GIT_SOURCES + BOR_SOURCES_DIR; README import section rewritten.
• uv run pytest green; uv run pytest --cov=app --cov-report=term-missing TOTAL ≥ pre-change number.
• uv run ruff check . && uv run pyright clean.
• .agent/user_stories/git-sources.md exists.
• One --no-gpg-sign commit staging only this phase's files; .agent/phases/todo/28_git_based_sources/ moved to .agent/phases/complete/.

Locked decisions

• A13 untouched — no DB migration; the importer's documents/chunks schema is unchanged (source/path are stored as before, keyed by the repo-relative path).
• A11 untouched — no new Python packages; git CLI is assumed available (standard on homelab machines). The clone/pull is done via subprocess (stdlib).
• A9 untouched — the A9 format filter / hidden-dir skip / exclusion list still apply to the cloned content (the importer's iter_importable_files is unchanged).
• No anchor revised — this is an operator-workflow change (how sources get onto disk), not a product/anchor change. Recorded in §11 + .env.example, not in §2.
• A16 honoured — unit + integration tests for the new logic; no story E2E required (no UI/API change).
• A17 honoured — one atomic --no-gpg-sign commit.