refactor(agents): migrate .agent/ planning tree to .agents/
Standardize on the .agents/ directory (shared with project skills): phases/, user_stories/, reports/, screenshots/, validate.sh, and phase-sessions/ + pipeline.log all move to .agents/ (git mv preserves history; runtime artifacts move alongside). Updates every reference in AGENTS.md, README.md, .gitignore, app docstrings, and test story headers. Historical KB content in data/ and the runtime pipeline.log transcript are left untouched.
This commit is contained in:
@@ -0,0 +1,41 @@
|
||||
# Phase 28 — Git-Based Sources
|
||||
|
||||
**Source:** `TODO.md L5 — "We shouldn't be hard-coding Homelab and Deployments. Instead, a list of git links should be specified. import_docs should clone or pull to a dedicated repository location and then index all the specified repository code."`
|
||||
**Story:** `.agents/user_stories/git-sources.md`
|
||||
**Context:** Phase 11 / the README document the import workflow. `scripts/import_docs.py` currently defaults to the hardcoded `DEFAULT_SOURCES = [Path("~/Homelab"), Path("~/Deployments")]` (repeatable via `--source`). The importer (`app/rag/importer.py::import_sources`) already walks any list of local directories — it needs no change to the walking logic, only to receive the resolved local paths.
|
||||
|
||||
## Objective
|
||||
Replace the hardcoded `~/Homelab` + `~/Deployments` default with a **list of git repository URLs** (`BOR_GIT_SOURCES`). `import_docs` clones (first run) or pulls (subsequent runs) each repository into a dedicated local location (`BOR_SOURCES_DIR`, default `~/bor-sources`) and indexes the resulting directories. The existing `--source` flag still overrides for manual paths.
|
||||
|
||||
## Dependencies
|
||||
- `01_infrastructure` (complete) — the base app + importer.
|
||||
- `11_long_answers` and the README import workflow (complete) — the `import_sources` contract and the documented CLI.
|
||||
|
||||
## Tasks
|
||||
1. `01_env_var_and_settings.md` — add `BOR_GIT_SOURCES` (comma-separated git URLs) + `BOR_SOURCES_DIR` settings; update `.env.example` + README.
|
||||
2. `02_git_clone_pull_utility.md` — create `scripts/git_sync.py` with `clone_or_pull(url, dest)` (clone --depth 1 / pull --ff-only, auth via URL or SSH).
|
||||
3. `03_import_docs_script_refactor.md` — refactor `scripts/import_docs.py` to resolve git sources → local dirs, then pass to `import_sources`.
|
||||
4. `04_integration_test_and_docs.md` — integration test for the git flow (mocked `git`); README + `.env.example` finalised.
|
||||
|
||||
## Testing & Quality
|
||||
- Unit/integration: `scripts/git_sync.py` `clone_or_pull` (clone when missing, pull when present, error propagation) — unit test with a mocked `subprocess`; `import_docs` resolves git URLs → local dirs (integration).
|
||||
- Coverage: **>90%** on `app/` (the importer is unchanged; the new logic is in `scripts/`, covered by its own tests).
|
||||
- E2E: none required (no `app/` or UI change) — but the smoke suite must stay green.
|
||||
|
||||
## Completion Criteria
|
||||
- [ ] With `BOR_GIT_SOURCES` set and `BOR_SOURCES_DIR` writable, `uv run python -m scripts.import_docs` clones (first run) or pulls (subsequent runs) each repo into `BOR_SOURCES_DIR/<name>/` and indexes them.
|
||||
- [ ] `--source <path>` still overrides to import an arbitrary local directory (unchanged behaviour).
|
||||
- [ ] Unreachable/invalid git URL → the script fails loudly (non-zero exit) naming the repo, without importing partial junk.
|
||||
- [ ] `.env.example` documents `BOR_GIT_SOURCES` + `BOR_SOURCES_DIR`; README import section rewritten.
|
||||
- [ ] `uv run pytest` green; `uv run pytest --cov=app --cov-report=term-missing` TOTAL ≥ pre-change number.
|
||||
- [ ] `uv run ruff check . && uv run pyright` clean.
|
||||
- [ ] `.agents/user_stories/git-sources.md` exists.
|
||||
- [ ] One `--no-gpg-sign` commit staging only this phase's files; `.agents/phases/todo/28_git_based_sources/` moved to `.agents/phases/complete/`.
|
||||
|
||||
## Locked decisions
|
||||
- **A13 untouched** — no DB migration; the importer's `documents`/`chunks` schema is unchanged (source/path are stored as before, keyed by the repo-relative path).
|
||||
- **A11 untouched** — no new Python packages; `git` CLI is assumed available (standard on homelab machines). The clone/pull is done via `subprocess` (stdlib).
|
||||
- **A9 untouched** — the A9 format filter / hidden-dir skip / exclusion list still apply to the cloned content (the importer's `iter_importable_files` is unchanged).
|
||||
- **No anchor revised** — this is an operator-workflow change (how sources get onto disk), not a product/anchor change. Recorded in §11 + `.env.example`, not in §2.
|
||||
- **A16 honoured** — unit + integration tests for the new logic; no story E2E required (no UI/API change).
|
||||
- **A17 honoured** — one atomic `--no-gpg-sign` commit.
|
||||
Reference in New Issue
Block a user