From 94d72285102536b641c1e127d0db1a2374845754 Mon Sep 17 00:00:00 2001 From: ducoterra Date: Thu, 27 Aug 2026 01:04:16 -0400 Subject: [PATCH] =?UTF-8?q?feat(admin):=20local=20directory=20sources=20?= =?UTF-8?q?=E2=80=94=20kind/path=20on=20git=5Fsources,=20combined=20sync?= =?UTF-8?q?=20+=20import,=20page=20form=20+=20badges?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit An existing, non-git directory is now a first-class source alongside the git repos: one table (git_sources + kind discriminator — A13 reversible migration), one admin page, one Sync button (phase locked decisions; the phase-35 table is extended, not duplicated). The DB is the local-source registry — no env var for local paths; BOR_GIT_SOURCES stays a git-only empty-table fallback. Migration 0007 (reversible, up/down integration-tested): git_sources.kind TEXT NOT NULL DEFAULT 'git' + ck_git_sources_kind (kind IN ('git','local')); git_sources.path TEXT NULL + uq_git_sources_path (mirrors 0006's uq_git_sources_url). Existing rows read kind='git', path=NULL. API (phase-35 contract extended, git byte-identical): POST kind=local requires path — trimmed, ~-expanded, absolute + an existing server directory, else 422 naming the path (fail loud at add-time); duplicate path 409 (named); wrong field combos 422. GET rows carry kind + path (git and env rows: path null); anonymous still 403 on every route (A10). Sync + import_docs resolve DB git + local rows together: git → clone_or_pull (unchanged); local → re-verified .is_dir() AT SYNC TIME (it may have moved/deleted since add-time) — a missing dir raises "local source missing: " (sanitized) before anything imports; one import_sources(..., prune=True) over the single combined list (pruning covers the union). Both-empty fails loudly ("no sources configured (git or local)"); --source still wins; the env fallback stays git-only. Page: second "Add a local directory" form (the same §7.4 never-stale button + inline-error lifecycle as the git form; 422/409 details name the path), Git/Local badges on rows (text + color, never color alone — WCAG), updated hint (git + local together, union prune); the anonymous sign-in gate is unchanged. Tests: 0007 up/down; the API local-kind matrix (403/201/422/409) with the git-kind suite green unchanged; the sync pipeline local/git/ mixed/missing against a host temp dir (the KB actually updated); import_docs DB resolution + --source precedence. Story E2E (isolated, deterministic across runs): add (Local badge) → missing path inline 422 naming it / duplicate 409 → the real Sync button imports the fixture file (GET /api/docs + sentinel in its content) → file deleted + sync prunes it (union prune) → row removed; anonymous gate + 403s (phase-35 regression). test_git_sources_admin.py (phase 35) green UNCHANGED — no selector collision with the new form; test_sync_button.py green. Docs: README — the two managed kinds (git = clone/pull mirror; local = direct in-place walk), add-time validation, union pruning, "the DB is the local-source registry (no env var for local paths)"; .env.example — the env fallback is git-only. --- .../01_kind_column.md | 24 + .../02_api_local_kind.md | 27 + .../03_sync_import_local.md | 30 ++ .../04_admin_page_local.md | 24 + .env.example | 10 +- README.md | 96 +++- alembic/versions/0007_git_sources_kind.py | 49 ++ app/api/git_sources.py | 157 ++++-- app/api/sync.py | 76 ++- app/models.py | 25 +- app/rag/git_sources.py | 65 ++- app/schemas.py | 63 ++- frontend/assets/git-sources.js | 172 ++++-- frontend/assets/styles.css | 80 ++- frontend/git-sources.html | 51 +- scripts/import_docs.py | 83 ++- tests/e2e/test_local_directory_sources.py | 496 ++++++++++++++++++ tests/integration/test_git_sources_api.py | 262 ++++++++- tests/integration/test_import_docs_git.py | 147 +++++- tests/integration/test_migration_0007.py | 261 +++++++++ tests/integration/test_sync_api.py | 204 ++++++- tests/unit/test_git_sources.py | 111 +++- 22 files changed, 2190 insertions(+), 323 deletions(-) create mode 100644 .agent/phases/complete/38_local_directory_sources/01_kind_column.md create mode 100644 .agent/phases/complete/38_local_directory_sources/02_api_local_kind.md create mode 100644 .agent/phases/complete/38_local_directory_sources/03_sync_import_local.md create mode 100644 .agent/phases/complete/38_local_directory_sources/04_admin_page_local.md create mode 100644 alembic/versions/0007_git_sources_kind.py create mode 100644 tests/e2e/test_local_directory_sources.py create mode 100644 tests/integration/test_migration_0007.py diff --git a/.agent/phases/complete/38_local_directory_sources/01_kind_column.md b/.agent/phases/complete/38_local_directory_sources/01_kind_column.md new file mode 100644 index 0000000..50816f7 --- /dev/null +++ b/.agent/phases/complete/38_local_directory_sources/01_kind_column.md @@ -0,0 +1,24 @@ +# Task 01 — `git_sources.kind` + `path` (migration 0007) + +**Phase:** `38_local_directory_sources` · **Source:** `TODO.md:11 — "Also need a way to import from existing directory if it's not a git repo"` +**Story:** `.agent/user_stories/local-directory-sources.md` + +## Objective +Extend the phase-35 `git_sources` table with a source-kind discriminator (`git` | `local`) and an optional local path, via a reversible migration — the foundation for the API, pipeline, and page tasks. + +## Work +1. `alembic/versions/0007_git_sources_kind.py` — read `alembic/versions/0006_git_sources.py` first and chain from its **actual** `revision` id (filenames are not revision ids — the phase-35 task-01 convention): + - `upgrade()`: `ALTER TABLE git_sources ADD COLUMN kind TEXT NOT NULL DEFAULT 'git'` + `ADD CONSTRAINT ck_git_sources_kind CHECK (kind IN ('git', 'local'))`; `ALTER TABLE git_sources ADD COLUMN path TEXT`; a unique index on `path` — a partial unique index (`WHERE path IS NOT NULL`) where the 0006 style allows it, otherwise a plain unique index (Postgres treats NULLs as distinct, and the API enforces local-only paths anyway — mirror whatever 0006 chose for `url`). + - `downgrade()`: drop the index, constraint, and columns in reverse. +2. `app/models.py` — extend the phase-35 `GitSource` model: `kind: Mapped[str]` (default `"git"`) + `path: Mapped[str | None]` (nullable, unique) and update the docstring (phase 38: kind discriminator; local rows carry `path`, git rows keep `url`). +3. Apply to the dev database: `podman compose up -d db` (if needed) then `uv run alembic upgrade head`; verify the columns + constraint + that existing rows read as `kind='git'`, `path=NULL`. +4. Migration test — follow the 0004/0005/0006 pattern in `tests/integration/`: up adds the columns/constraint/index (existing rows keep `kind='git'`), down drops them (round-trip on the test DB). + +## Testing & Quality +- Integration: the up/down test above; full `uv run pytest` green. +- Coverage: model + migration only — the `app/` gate stays >90%. + +## Completion Criteria +- [ ] `alembic/versions/0007_git_sources_kind.py` chains off 0006's real revision id; up/down reversible. +- [ ] `uv run alembic upgrade head` applies cleanly; existing rows read `kind='git'`, `path=NULL`. +- [ ] The 0007 up/down integration test passes; full `uv run pytest` green; `uv run ruff check . && uv run pyright` clean. diff --git a/.agent/phases/complete/38_local_directory_sources/02_api_local_kind.md b/.agent/phases/complete/38_local_directory_sources/02_api_local_kind.md new file mode 100644 index 0000000..05e833a --- /dev/null +++ b/.agent/phases/complete/38_local_directory_sources/02_api_local_kind.md @@ -0,0 +1,27 @@ +# Task 02 — The admin API: the local kind + +**Phase:** `38_local_directory_sources` · **Source:** `TODO.md:11 — "Also need a way to import from existing directory if it's not a git repo"` +**Story:** `.agent/user_stories/local-directory-sources.md` + +## Objective +Extend the phase-35 git-sources CRUD to accept and return `kind=local` rows with fail-loud path validation, keeping the git contract byte-identical. + +## Work +1. The phase-35 API module (the file behind `/api/git-sources` — identify it in the repo; phase 35 created it): + - `POST /api/git-sources` body: `{kind?: "git"|"local" (default "git"), url?, path?}`: + - `kind=git` → exactly today's `url` validation (trimmed, 1–500 chars, the `https?://` / `ssh://` / `git@` shape, 409 on duplicate without echoing the URL). + - `kind=local` → `path` required: trimmed; `Path(p).expanduser()`; must be **absolute after expansion** and an **existing directory on the server** → else `422 {detail: "local source path is not a directory: "}` (a missing path is a user error — fail loud at add-time so the owner sees it immediately; the path is not a secret, so echo it); 409 on a duplicate `path` (detail may name the path). + - Wrong field combinations (git without url, local without path, both kinds' fields) → 422. + - `GET /api/git-sources` rows gain `kind` + `path` (git rows: `path: null`; the env-fallback rows report `kind: "git"`; `from_env: true` semantics unchanged — env rows are git-only). + - `DELETE /api/git-sources/{id}` — unchanged (removal prunes on the next sync, as today). +2. The phase-35 request/response models (wherever they live — `app/schemas.py` or the module) — extend for the new fields; keep the OpenAPI docs accurate. +3. Integration tests (extend `tests/integration/test_git_sources_api.py`): anonymous → 403 on all routes (regression); `kind=local` + an existing temp dir → 201 with the stored row (`kind=local`, `path` stored expanded); a relative path → 422; a missing path → 422 naming the path; a duplicate path → 409; a git row still validates exactly as before (regression); GET mixes kinds in added order. + +## Testing & Quality +- Integration: the matrix above; the existing git-kind suite green unchanged. +- Coverage: **>90%** on the modified module. + +## Completion Criteria +- [ ] The local-kind POST matrix (201 / 422 / 409) green; the git contract unchanged. +- [ ] GET rows carry `kind` + `path`; the env fallback stays git-only. +- [ ] Full `uv run pytest` green; `uv run ruff check . && uv run pyright` clean. diff --git a/.agent/phases/complete/38_local_directory_sources/03_sync_import_local.md b/.agent/phases/complete/38_local_directory_sources/03_sync_import_local.md new file mode 100644 index 0000000..f69604a --- /dev/null +++ b/.agent/phases/complete/38_local_directory_sources/03_sync_import_local.md @@ -0,0 +1,30 @@ +# Task 03 — Sync + `import_docs`: git and local together + +**Phase:** `38_local_directory_sources` · **Source:** `TODO.md:11 — "Also need a way to import from existing directory if it's not a git repo"` +**Story:** `.agent/user_stories/local-directory-sources.md` + +## Objective +The canonical mirror action (the Sync button) and the CLI import DB **git + local** rows in one run: git rows clone/pull as today, local rows are walked directly; a missing local directory aborts the run loudly before anything is imported. + +## Work +1. The phase-35 resolution module (`app/rag/git_sources.py` or wherever `effective_git_sources()` lives — identify it in the repo) — extend: + - `effective_sources(db) -> list[GitSource]` — the DB rows of **both** kinds (DB-wins / env-fallback semantics unchanged: while the table is empty, the env git URLs surface as synthetic `kind='git'` rows with `from_env`). Keep a backward-compatible alias (`effective_git_sources`) if other modules import the old name. + - Log the list origin as today (`origin=db|env`) plus the kind counts (`git=N local=M`). +2. `app/api/sync.py::_run_sync`: + - For each resolved row: `kind=git` → `clone_or_pull(url, sources_root / repo_name(url))` (unchanged); `kind=local` → `Path(row.path).expanduser()`, verify `.is_dir()` **at sync time** (the directory may have moved/deleted since add-time) → else raise a sync error `local source missing: ` (no credentials involved, but run it through the existing `_sanitize_error` for consistency). + - `import_sources(combined, llm, prune=True)` over the **single combined list** (git checkouts + local dirs) — pruning covers the union (phase-32 semantics); the overview regeneration is unchanged (change-gated). + - The fail-loud empty-config check: no git rows, no local rows, and no env URLs → `"no sources configured (git or local)"` (replaces phase 32's git-only message). +3. `scripts/import_docs.py` — `_resolve_sources` gains the DB path (after `--source`, which still wins over everything): no `--source` and the DB table non-empty → the combined list (git cloned/pulled + local direct); table empty → env git URLs (as today) → the legacy `DEFAULT_SOURCES`. A missing local dir → abort with the path named, before importing anything (the same pre-import fail-loud as a failing git clone). +4. Unit/integration: + - Unit: `effective_sources` — mixed kinds, DB-wins, env fallback git-only, both-empty (extend the phase-35 tests). + - Integration (the `test_sync_api.py` pattern, with a **temp local dir** — a host temp dir containing one fixture `.md`, since the app server runs on the same host): local-only, git-only, and mixed syncs — the local file lands in the KB (`GET /api/docs` as admin); a missing local path → status `failed` with `local source missing: …` in the sanitized error; `import_docs` (no `--source`) with a DB local row imports it; `--source` still wins over the DB. + +## Testing & Quality +- The suites above; `test_git_sources_api.py`, `test_sync_api.py`, `test_import_docs_git.py` (phase 35's list) stay green through the indirection. +- Coverage: **>90%** on the modified modules. + +## Completion Criteria +- [ ] A mixed git + local sync imports both in one run; union pruning works (a file deleted from the local dir is pruned on the next sync). +- [ ] A missing local dir fails the run loudly (the status error names the path) and imports nothing. +- [ ] `import_docs`: DB git + local resolution; `--source` wins; the env fallback is git-only; the both-empty message is updated. +- [ ] Full `uv run pytest` green; coverage gate holds; `uv run ruff check . && uv run pyright` clean. diff --git a/.agent/phases/complete/38_local_directory_sources/04_admin_page_local.md b/.agent/phases/complete/38_local_directory_sources/04_admin_page_local.md new file mode 100644 index 0000000..88bd627 --- /dev/null +++ b/.agent/phases/complete/38_local_directory_sources/04_admin_page_local.md @@ -0,0 +1,24 @@ +# Task 04 — The page: the local-directory form + badges + +**Phase:** `38_local_directory_sources` · **Source:** `TODO.md:11 — "Also need a way to import from existing directory if it's not a git repo"` +**Story:** `.agent/user_stories/local-directory-sources.md` + +## Objective +The sources page (phase 35) can add a **local directory** next to git repos, the list shows which is which, and the hint reflects the combined Sync semantics. + +## Work +1. The phase-35 page (the git-sources HTML template + its page script — identify the actual paths in the repo): + - A second add form, **"Local directory"**: a labeled text input (placeholder `~/Notes`), an Add button, an inline error slot — the same never-stale-button + inline-error pattern as the git form (PLAN §7.4); on success the form clears and the list re-fetches. + - List rows: a kind badge — `Git` / `Local` (a small styled span, distinguishable by color **and** text, not color alone — WCAG) — plus the mono value (git URL as today; the full local path) + added date + Remove (Remove is unchanged — it prunes on the next sync). + - The hint text: "Sync clones/pulls the git repos and imports the local directories together (files removed from a source are pruned)." + - Anonymous: the sign-in gate unchanged; the admin-only nav link unchanged (phase 35/29). +2. `frontend/assets/styles.css` — the badge styles + the second form (reuse the existing form styles; contrast ≥ 4.5:1 in both themes). +3. UI Structure Check (AGENTS.md rule 5) before finalizing: landmarks intact, both inputs labeled, focus-visible on the new Add button, the page stays inside the shared layout (no new top-level structure). + +## Testing & Quality +- Gated by the story E2E (task 05); no CDN (AGENTS.md rule 6 — no new external tags). + +## Completion Criteria +- [ ] The admin adds a local directory through the page; the row appears with the Local badge; an invalid path shows the 422 detail inline and the button recovers (never stale). +- [ ] The git form + all existing page behavior unchanged. +- [ ] `uv run pytest` green (no frontend unit layer — the E2E is the gate); `uv run ruff check . && uv run pyright` clean. diff --git a/.env.example b/.env.example index b33fe14..c3a2894 100644 --- a/.env.example +++ b/.env.example @@ -55,8 +55,14 @@ BOR_AGENT_READ_CALLS=1 # per-turn read_document opportunities (0 disa # variable is the EMPTY-TABLE FALLBACK: it only applies while the admin # list is empty; once the page has stored any source, this variable is # ignored (the page is the source of truth). Empty table + empty variable -# = no git sources (import_docs falls back to --source / the old -# ~/Homelab + ~/Deployments defaults; the UI Sync button fails loudly). +# + no local rows = no sources (import_docs falls back to --source / the +# old ~/Homelab + ~/Deployments defaults; the UI Sync button fails loudly +# with "no sources configured (git or local)"). +# +# Phase 38: this env fallback is GIT-ONLY. Local directory sources (an +# existing, non-git directory on the server) have NO env var — the DB is +# the local-source registry: add them on the same admin page (the +# "Add a local directory" form, kind 'local' + path, migration 0007). # BOR_GIT_SOURCES=https://github.com/user/homelab.git,https://github.com/user/deployments.git # BOR_SOURCES_DIR=~/bor-sources diff --git a/README.md b/README.md index 86e48fb..672beb0 100644 --- a/README.md +++ b/README.md @@ -63,23 +63,29 @@ uv run python -m scripts.llm_probe # sanity: models + 768-dim check uv run python -m scripts.import_docs # import the configured sources (below) ``` -Two source modes: +Two managed source kinds (one page, one registry) plus a manual override: -- **Git sources (recommended)** — the list is managed on the **admin - Git sources page** (`/git-sources.html`) and stored in Postgres (see +- **Git sources** — managed on the **admin Git sources page** + (`/git-sources.html`) and stored in Postgres (see [Git-based sources](#git-based-sources)). `import_docs` clones each repo (first run) or pulls it (subsequent runs) into `BOR_SOURCES_DIR//` (default `~/bor-sources`) and indexes the checkouts. While the stored list is empty, the `BOR_GIT_SOURCES` variable in `.env` is the fallback — the moment the page stores a source, the variable is ignored. +- **Local directory sources** (phase 38) — an existing, non-git directory + on the server, registered on the *same* admin page (see + [Local directory sources](#local-directory-sources)). No clone, no + checkout copy: the directory is walked in place. There is **no env var + for local paths** — the DB is the registry. - **Manual directories** — `--source ` (repeatable) imports local - directories directly and *always wins* over the git sources (stored list - or env). + directories directly and *always wins* over the stored sources (git and + local) and the env fallback. - If neither is set (stored list, `--source`, and `BOR_GIT_SOURCES` all - empty), the import falls back to the **previous** default, + empty), `import_docs` falls back to the **previous** default, `~/Homelab` + `~/Deployments` — kept only for backwards compatibility, - now replaced by the git sources list. + now replaced by the managed sources; the UI Sync button instead fails + loudly ("no sources configured (git or local)"). ### 6. Run the app ```bash @@ -107,12 +113,18 @@ uv run uvicorn app.main:app --reload new tab). **Admin-only** — anonymous visitors see a sign-in gate instead (the catalog is what the login locks; the document viewer itself stays open to everyone). -- **Git sources** (`/git-sources.html`) — the list of git repositories the - **Sync sources** button clones and indexes; **admin-only** (the same - sign-in gate as Sources). Add or remove repositories here — no `.env` - editing, no restart. Adding/removing does not clone or prune on its - own: the Sync button performs that, and a removed repository's documents - leave the index on the next sync. +- **Git sources** (`/git-sources.html`) — the admin-managed source + registry: the git repositories the **Sync sources** button clones and + indexes, **and** existing local directories it imports directly + (phase 38 — one table with a `kind` discriminator, one page); **admin-only** + (the same sign-in gate as Sources). Add or remove sources here — no + `.env` editing, no restart. A local directory must be an absolute, + existing directory at add-time (a missing/relative path is rejected + inline, naming the path; so are duplicates); list rows carry a **Git** + or **Local** badge. Adding/removing does not clone or prune on its + own: the Sync button performs that (git + local together, one run, + prune over the union), and a removed source's documents leave the index + on the next sync. ## Thinking @@ -345,16 +357,52 @@ BOR_SOURCES_DIR=~/bor-sources # default; each repo lands in // **nothing** (no partial junk). Fix the URL/connectivity and re-run — the other checkouts stay on disk and are pulled as usual. +### Local directory sources + +Not every set of notes lives in a git repo — a plain directory can be a +first-class source too (phase 38). It shares the git sources' **one +table** (the `git_sources` registry with a `kind` discriminator: `git` | +`local`, migration 0007), **one admin page**, and **one Sync button**: + +- **Add it on the Git sources page** — the “Add a local directory” form + next to the git form. Add-time validation fails loud: the path is + trimmed, `~` is expanded, and must be an **absolute, existing directory + on the server** — anything else (missing, relative, a file) is rejected + with the path named inline; a duplicate path is rejected the same way. + There is **no env var for local paths** — the DB is the local-source + registry (`BOR_GIT_SOURCES` stays a git-only fallback). +- **Sync walks it directly** — no clone, no checkout copy: each run + indexes the directory in place (A9 format filter, hidden-dir skip, + sha256 delta), together with the git checkouts in the **same run**. + `documents.source` is the directory's name. The directory is + re-verified to exist **at sync time** (it may have moved or been + deleted since add-time): a missing directory fails the run loudly, + naming the path, and imports **nothing** (the same pre-import fail-loud + as a failing git clone). +- **Pruning is over the union** — git checkouts and local directories are + imported together with `prune=True`, so a file removed from a local + directory, a repo, or a removed source leaves the index on that run. + Removing the row on the page stops the directory being a source; its + documents leave the index on the next sync (exactly like git sources). +- **`import_docs`** (no `--source`) resolves the stored git **and** local + rows — git cloned/pulled as above, local walked directly — in one run; + `--source` still wins over everything; while the table is empty, + `BOR_GIT_SOURCES` is the git-only fallback; no git rows, no local rows, + and no env URLs fails loudly ("no sources configured (git or local)"). + ### Sync from the UI The **Sync sources** button on the **Sources** page — visible to the **admin only** (anonymous visitors never see it) — runs the whole git-source refresh in one click, in-process: -1. **clone/pull** every configured git source — the admin-managed list - (the `git_sources` table; `BOR_GIT_SOURCES` only while that list is - empty), through the same `clone_or_pull` the CLI uses (shallow clone - on first run, `git pull --ff-only` afterwards); +1. **clone/pull + walk** every configured source — the git sources (the + admin-managed `git_sources` table; `BOR_GIT_SOURCES` only while that + list is empty) through the same `clone_or_pull` the CLI uses (shallow + clone on first run, `git pull --ff-only` afterwards), **and** the + local directories registered on the same page, walked directly + (re-verified to exist at sync time — a missing directory fails the run + loudly, naming the path); 2. **re-import with prune** — the `--prune` equivalent, so files deleted upstream leave the index (the button is the canonical "mirror the repos" action); the sha256 delta still skips unchanged files, so an @@ -363,12 +411,12 @@ git-source refresh in one click, in-process: chat turn injects) — but only when the import actually changed the knowledge base. -- **Prerequisites:** at least one git source must be configured — a row - on the admin Git sources page, or `BOR_GIT_SOURCES` in `.env` while the - stored list is empty; **both** empty fails the sync loudly ("no git - sources configured"), because the button targets the git repos only - (manual `--source` directories have no repo to clone) — and `git` must - be on the app's `PATH`. +- **Prerequisites:** at least one source must be configured — a git or + local row on the admin Git sources page, or `BOR_GIT_SOURCES` in `.env` + while the stored list is empty (git-only); **all** empty fails the sync + loudly ("no sources configured (git or local)"), because the button + targets the admin-managed registry (manual `--source` directories have + no place in it) — and `git` must be on the app's `PATH` for git sources. - **States:** clicking starts the run (`202`) and the button goes disabled with **Syncing…** (spinning icon) while the page polls `GET /api/sync/status` every 2 s. There is deliberately **no @@ -597,7 +645,7 @@ served locally (no CDN), `BOR_ENVIRONMENT=production`. | `BOR_AGENT_LIST_CALLS` | `1` | per-turn `list_documents` tool opportunities on grounded turns (0 disables the tool) | | `BOR_AGENT_READ_CALLS` | `1` | per-turn `read_document` tool opportunities on grounded turns (0 disables the tool) | | `BOR_IMPORT_EXTENSIONS` | `md,markdown,txt,yaml,yml,json,py` | csv of importable formats (may only narrow the A9 set) | -| `BOR_GIT_SOURCES` | — (empty) | csv of git repo URLs — **fallback while the admin Git sources page's list (Postgres `git_sources`) is empty**; the page is the primary management surface (see *Git-based sources*) | +| `BOR_GIT_SOURCES` | — (empty) | csv of git repo URLs — **fallback while the admin Git sources page's list (Postgres `git_sources`) is empty**; the page is the primary management surface (see *Git-based sources*). **Git-only**: local directory sources have no env var — they are registered on the admin page (see *Local directory sources*) | | `BOR_SOURCES_DIR` | `~/bor-sources` | where the git source repos are cloned/pulled (one subdirectory per repo) | | `BOR_STEERING_MAX_CHARS` | `8000` | char budget for the `` (steering notes) prompt section | | `BOR_SUMMARY_MAX_CHARS` | `12000` | cap on document content sent to the `lite` summary model at import (see *Document summaries*) | diff --git a/alembic/versions/0007_git_sources_kind.py b/alembic/versions/0007_git_sources_kind.py new file mode 100644 index 0000000..2ace686 --- /dev/null +++ b/alembic/versions/0007_git_sources_kind.py @@ -0,0 +1,49 @@ +"""git_sources.kind + path: local-directory source discriminator + +Revision ID: 0007 +Revises: 0006 +Create Date: 2026-08-26 + +Phase 38 (local-directory-sources story, A13 — additive, reversible +columns; the phase-35 table is extended, not duplicated): + +* ``git_sources.kind`` — TEXT NOT NULL with server default ``'git'`` and + check constraint ``ck_git_sources_kind`` (``kind IN ('git', 'local')``). + Existing rows read as ``kind='git'``; git rows keep ``url``, local + rows carry an existing directory under ``path`` (walked directly — + no clone/pull). +* ``git_sources.path`` — TEXT, nullable (NULL for git rows). Made unique + by the plain unique index ``uq_git_sources_path`` — mirrors 0006's + ``uq_git_sources_url`` choice for ``url`` (Postgres treats NULLs as + distinct under a unique index, and the API enforces local-only paths + anyway). +""" +from __future__ import annotations + +import sqlalchemy as sa + +from alembic import op + +revision = "0007" +down_revision = "0006" +branch_labels = None +depends_on = None + + +def upgrade() -> None: + op.add_column( + "git_sources", + sa.Column("kind", sa.Text(), server_default=sa.text("'git'"), nullable=False), + ) + op.create_check_constraint( + "ck_git_sources_kind", "git_sources", "kind IN ('git', 'local')" + ) + op.add_column("git_sources", sa.Column("path", sa.Text(), nullable=True)) + op.create_index("uq_git_sources_path", "git_sources", ["path"], unique=True) + + +def downgrade() -> None: + op.drop_index("uq_git_sources_path", table_name="git_sources") + op.drop_constraint("ck_git_sources_kind", table_name="git_sources", type_="check") + op.drop_column("git_sources", "path") + op.drop_column("git_sources", "kind") diff --git a/app/api/git_sources.py b/app/api/git_sources.py index 3fb9c94..f08c8f6 100644 --- a/app/api/git_sources.py +++ b/app/api/git_sources.py @@ -1,33 +1,42 @@ -"""Admin-managed git sources API (phase 35, task 02). +"""Admin-managed sources API (phase 35, task 02; local kind, phase 38). Admin-only CRUD under ``/api/git-sources`` (phase 16 pattern, A10 extended — the public API surface stays stateless and the signed cookie remains the only session state, same as ``/api/steering`` and -``/api/sync``): the ``git_sources`` table holds the repo URLs the Sync -button (phase 32) and ``import_docs`` (phase 28) clone/pull. DB rows win -over ``BOR_GIT_SOURCES``, which is a fallback while the table is empty -(the phase's locked decision — ``from_env`` tells the UI which list it is -looking at, so the page can show the env note only while the fallback is -active). +``/api/sync``): the ``git_sources`` table holds the sources the Sync +button (phase 32) and ``import_docs`` (phase 28) import — ``kind='git'`` +rows carry the repo URL to clone/pull, ``kind='local'`` rows (phase 38) +carry an existing directory on the server to walk directly. DB rows win +over ``BOR_GIT_SOURCES``, which is a git-only fallback while the table is +empty (the phase's locked decision — ``from_env`` tells the UI which list +it is looking at, so the page can show the env note only while the +fallback is active). Routes: ``GET`` (DB rows oldest-first, or the env list with -``from_env: true`` while the table is empty), ``POST`` (201, validated -create), ``DELETE /{source_id}`` (204). The whole router sits behind -:func:`app.core.auth.require_admin` — anonymous callers get 403 on every -route. +``from_env: true`` while the table is empty; rows carry ``kind`` + +``path``, git rows — and env rows — report ``path: null``), ``POST`` +(201, validated create; ``kind`` selects the validation: git → exactly +the phase-35 URL contract, local → an existing absolute directory, else +422 naming the path), ``DELETE /{source_id}`` (204). The whole router +sits behind :func:`app.core.auth.require_admin` — anonymous callers get +403 on every route. -No credential-echo path: URLs may embed ``user:pass@`` (phase 32's -masking discipline), so the 409/422 details are fixed generic strings -that never repeat the submitted URL. +No credential-echo path: git URLs may embed ``user:pass@`` (phase 32's +masking discipline), so every git 409/422 detail is a fixed generic +string that never repeats the submitted URL. Local paths are not +secrets — the local 422/409 details name the (expanded) path so the +owner sees exactly which directory failed. -Scope boundary (phase locked decisions): adding or removing a repo does -NOT clone, import, or prune anything — the existing Sync button performs -that (a removal prunes on the next sync, ``prune=True``). +Scope boundary (phase locked decisions): adding or removing a source +does NOT clone, import, or prune anything — the existing Sync button +performs that (a removal prunes on the next sync, ``prune=True``). """ from __future__ import annotations import re import uuid +from pathlib import Path +from typing import Literal, cast from fastapi import APIRouter, Depends, HTTPException, Response from sqlalchemy import select @@ -38,7 +47,7 @@ from app.config import get_settings from app.core.auth import require_admin from app.db import get_db from app.models import GitSource -from app.schemas import GitSourceIn, GitSourceList, GitSourceOut +from app.schemas import GitSourceIn, GitSourceList, GitSourceOut, GitSourceRow router = APIRouter( prefix="/git-sources", @@ -57,24 +66,37 @@ URL_RE = re.compile(r"^(https?://|ssh://|git@)") def list_git_sources( db: Session = Depends(get_db), # noqa: B008 ) -> GitSourceList: - """The effective git source list. + """The effective source list (git + local rows, phase 38). DB rows ordered by ``(added_at, id)`` (oldest first, id tie-break for - same-timestamp inserts) with ``from_env: false``; while the table is - empty, the ``BOR_GIT_SOURCES`` env URLs as rows with null - ``id``/``added_at`` and ``from_env: true``. + same-timestamp inserts) with ``from_env: false`` — each row carries + its ``kind`` and, for local rows, the stored ``path`` (git rows and + env rows report ``path: null``); while the table is empty, the + ``BOR_GIT_SOURCES`` env URLs as git rows (the env fallback is + git-only) with null ``id``/``added_at`` and ``from_env: true``. """ rows = db.scalars( select(GitSource).order_by(GitSource.added_at.asc(), GitSource.id.asc()) ).all() if rows: return GitSourceList( - sources=[GitSourceOut(id=row.id, url=row.url, added_at=row.added_at) for row in rows], + sources=[ + # ``ck_git_sources_kind`` (migration 0007) guarantees the + # value is 'git' or 'local' — the cast documents that. + GitSourceRow( + id=row.id, + kind=cast(Literal["git", "local"], row.kind), + url=row.url, + path=row.path, + added_at=row.added_at, + ) + for row in rows + ], from_env=False, ) return GitSourceList( sources=[ - GitSourceOut(id=None, url=url, added_at=None) + GitSourceRow(id=None, kind="git", url=url, path=None, added_at=None) for url in get_settings().git_source_list ], from_env=True, @@ -86,14 +108,49 @@ def create_git_source( payload: GitSourceIn, db: Session = Depends(get_db), # noqa: B008 ) -> GitSourceOut: - """Store one repo URL (already trimmed by the schema). + """Store one source (fields already trimmed by the schema). - 422 when the shape is not one of the accepted prefixes (generic - detail — the input is never echoed); 409 when the trimmed URL is - already stored (same; the unique index is the backstop against a - concurrent insert the pre-check missed); 201 + the created row - otherwise. + ``kind="git"`` (default) — exactly the phase-35 contract: 422 when + the URL shape is not one of the accepted prefixes (generic detail — + the input is never echoed), 409 when the trimmed URL is already + stored (the unique index is the backstop against a concurrent insert + the pre-check missed), 201 + the created row otherwise. + + ``kind="local"`` — ``path`` must expand (``~``) to an absolute, + existing directory on the server: 422 naming the path otherwise + (fail loud at add-time — the owner sees it immediately), 409 when + the path is already stored (detail names the path), 201 + the stored + row otherwise (``url`` holds the expanded path — the table's + NOT-NULL location column). + + Wrong field combinations (git without url, local without path, both + kinds' fields) are 422 with fixed, input-free details. """ + row = _create_git_row(payload, db) if payload.kind == "git" else _create_local_row(payload, db) + return GitSourceOut(id=row.id, url=row.url, added_at=row.added_at) + + +def _commit_new(row: GitSource, duplicate_detail: str, db: Session) -> GitSource: + """Insert ``row``; the unique index is the backstop — a concurrent + insert the pre-check missed still yields the generic 409, never a + 500 (phase-35 convention, now shared by both kinds).""" + db.add(row) + try: + db.commit() + except IntegrityError: + db.rollback() + raise HTTPException(status_code=409, detail=duplicate_detail) from None + db.refresh(row) + return row + + +def _create_git_row(payload: GitSourceIn, db: Session) -> GitSource: + """``kind=git`` — the phase-35 URL contract, unchanged (A10: no + credential echo, so every detail is a fixed string).""" + if payload.path is not None: + raise HTTPException(status_code=422, detail="a git source takes a url, not a path") + if payload.url is None: + raise HTTPException(status_code=422, detail="a git source requires a url") url = payload.url if not URL_RE.match(url): raise HTTPException( @@ -101,17 +158,37 @@ def create_git_source( ) if db.scalar(select(GitSource).where(GitSource.url == url)) is not None: raise HTTPException(status_code=409, detail="a git source with this URL already exists") - row = GitSource(url=url) - db.add(row) - try: - db.commit() - except IntegrityError: - db.rollback() + return _commit_new( + GitSource(url=url, kind="git"), "a git source with this URL already exists", db + ) + + +def _create_local_row(payload: GitSourceIn, db: Session) -> GitSource: + """``kind=local`` — fail-loud add-time validation (phase 38): + trimmed → ``expanduser()`` → absolute + existing directory, else 422 + naming the path (not a secret, unlike a git URL).""" + if payload.url is not None: + raise HTTPException(status_code=422, detail="a local source takes a path, not a url") + if payload.path is None: + raise HTTPException(status_code=422, detail="a local source requires a path") + expanded = Path(payload.path).expanduser() + if not expanded.is_absolute() or not expanded.is_dir(): raise HTTPException( - status_code=409, detail="a git source with this URL already exists" - ) from None - db.refresh(row) - return GitSourceOut(id=row.id, url=row.url, added_at=row.added_at) + status_code=422, detail=f"local source path is not a directory: {expanded}" + ) + path = str(expanded) + if db.scalar(select(GitSource).where(GitSource.path == path)) is not None: + raise HTTPException( + status_code=409, detail=f"a local source with this path already exists: {path}" + ) + # ``url`` is the table's NOT-NULL location column (phase 38: local + # rows carry the expanded path there too — git URL shapes and absolute + # paths cannot collide). + return _commit_new( + GitSource(url=path, kind="local", path=path), + f"a local source with this path already exists: {path}", + db, + ) @router.delete("/{source_id}", status_code=204) diff --git a/app/api/sync.py b/app/api/sync.py index cb7cdbe..4b6ef9e 100644 --- a/app/api/sync.py +++ b/app/api/sync.py @@ -12,20 +12,27 @@ plus a module-level :class:`SyncStatus` that the UI polls every 2 s 409; the status object is authoritative, so the UI can never sit on a stale button state (§7.4 adaptation, phase locked decisions). -Pipeline (the canonical "mirror the repos" action — phase locked +Pipeline (the canonical "mirror the sources" action — phase locked decisions): -1. resolve the effective git sources — the ``git_sources`` DB rows, - else the ``BOR_GIT_SOURCES`` fallback - (:func:`app.rag.git_sources.effective_git_sources`, shared with the - CLI) — empty on both origins fails loudly (``no git sources - configured``) instead of silently importing the legacy local - directories; -2. :func:`scripts.git_sync.clone_or_pull` each repo into - ``BOR_SOURCES_DIR//`` (phase 28 — reused, not - re-implemented; a failing repo aborts before any import); -3. ``import_sources(..., prune=True)`` — prune so files deleted - upstream leave the index (the CLI's no-prune default is unchanged); +1. resolve the effective sources — the ``git_sources`` DB rows (git + **and** local, phase 38), else the ``BOR_GIT_SOURCES`` fallback + (git-only) + (:func:`app.rag.git_sources.effective_sources`, shared with the + CLI) — empty on both origins (no git rows, no local rows, no env + URLs) fails loudly (``no sources configured (git or local)``) + instead of silently importing the legacy local directories; +2. per resolved row: ``kind=git`` → :func:`scripts.git_sync.clone_or_pull` + into ``BOR_SOURCES_DIR//`` (phase 28 — reused, not + re-implemented); ``kind=local`` → the stored directory, re-verified + ``.is_dir()`` **at sync time** (it may have moved/deleted since + add-time) — a missing directory raises ``local source missing: + ``; a failing clone or a missing local dir aborts before any + import; +3. ``import_sources(..., prune=True)`` over the single combined list + (git checkouts + local dirs) — prune so files deleted upstream or + out of a local dir leave the index (pruning covers the union; the + CLI's no-prune default is unchanged); 4. when the import changed the KB (added + updated > 0), ``regenerate_overview`` refreshes the single ``kb_overview`` row (phase 31 trigger, best-effort inside). @@ -48,7 +55,7 @@ from fastapi import APIRouter, Depends, HTTPException from app.config import get_settings from app.core.auth import require_admin from app.db import SessionLocal -from app.rag.git_sources import effective_git_sources +from app.rag.git_sources import effective_sources from app.rag.importer import ImportSummary, import_sources from app.rag.llm import LLMClient from app.rag.overview import regenerate_overview @@ -147,24 +154,41 @@ async def _run_sync() -> None: try: settings = get_settings() # The background task has no request session: open a short-lived - # one around the shared phase-35 resolver (DB rows win, the - # BOR_GIT_SOURCES list is a fallback while the table is empty). + # one around the shared phase-35/38 resolver (DB rows of both + # kinds win; the BOR_GIT_SOURCES git list is a fallback while + # the table is empty). db = SessionLocal() try: - git_urls, origin = effective_git_sources(db) + rows, origin = effective_sources(db) finally: db.close() - if not git_urls: - # The button targets git sources only (manual --source dirs - # have no repo to clone) — an empty config on *both* origins - # fails loudly instead of silently importing the legacy - # directories. - raise GitSyncError( - "no git sources configured (git_sources table empty and BOR_GIT_SOURCES unset)" - ) - logger.info("sync: started repos=%d origin=%s", len(git_urls), origin) + if not rows: + # The button targets the admin-managed source registry + # (manual --source dirs have no repo to clone) — an empty + # config on *both* origins (no git rows, no local rows, no + # env URLs) fails loudly instead of silently importing the + # legacy directories. + raise GitSyncError("no sources configured (git or local)") + git_count = sum(1 for row in rows if row.kind == "git") + logger.info( + "sync: started repos=%d origin=%s git=%d local=%d", + len(rows), origin, git_count, len(rows) - git_count, + ) sources_root = Path(settings.sources_dir).expanduser() - sources = [clone_or_pull(url, sources_root / repo_name(url)) for url in git_urls] + sources: list[Path] = [] + for row in rows: + if row.kind == "git": + sources.append(clone_or_pull(row.url, sources_root / repo_name(row.url))) + else: + # kind=local — the stored expanded path (phase 38 also + # mirrors it in the NOT-NULL ``url`` location column, the + # ``or`` keeps the type checker honest); re-verified at + # sync time because the directory may have moved or been + # deleted since add-time. + path = Path(row.path or row.url).expanduser() + if not path.is_dir(): + raise GitSyncError(f"local source missing: {path}") + sources.append(path) llm = LLMClient() summary: ImportSummary = await import_sources(sources, llm, prune=True) overview = False diff --git a/app/models.py b/app/models.py index 618d647..af80737 100644 --- a/app/models.py +++ b/app/models.py @@ -13,8 +13,10 @@ Data model — see ``.agent/PLAN.md`` §Data Model: * ``kb_overview`` — single-row lite-generated outline of the KB's basic categories, injected as the ```` section of every chat turn (phase 31). -* ``git_sources`` — admin-managed git source URLs the Sync button and - import_docs clone/pull (phase 35). +* ``git_sources`` — admin-managed source registry (git URLs + local + directories) the Sync button and import_docs + import (phase 35; ``kind`` discriminator added in + phase 38). """ from __future__ import annotations @@ -138,16 +140,25 @@ class KbOverview(Base): class GitSource(Base): - """One admin-managed git source (phase 35). + """One admin-managed source (phase 35; kind discriminator, phase 38). - The UI-maintained list of repo URLs the Sync button (phase 32) and - import_docs (phase 28) clone/pull. DB rows win over the - BOR_GIT_SOURCES env var, which is a fallback while this table is - empty (see app.rag.git_sources.effective_git_sources). + The UI-maintained list the Sync button (phase 32) and import_docs + (phase 28) import from. ``kind`` discriminates: ``git`` rows carry a + repo ``url`` (cloned/pulled), ``local`` rows carry an existing + directory ``path`` (walked directly). DB rows win over the + BOR_GIT_SOURCES env var (git-only fallback), which is a fallback + while this table is empty (see + app.rag.git_sources.effective_git_sources). """ __tablename__ = "git_sources" id: Mapped[uuid.UUID] = mapped_column(UUID(as_uuid=True), primary_key=True, default=uuid.uuid4) url: Mapped[str] = mapped_column(Text, unique=True, nullable=False) + #: Source-kind discriminator (phase 38): "git" (default) or "local" + #: — enforced by the ``ck_git_sources_kind`` CHECK constraint. + kind: Mapped[str] = mapped_column(Text, default="git", server_default="'git'") + #: Absolute directory of a ``local`` source; NULL for git rows. + #: Unique — Postgres treats NULLs as distinct under a unique index. + path: Mapped[str | None] = mapped_column(Text, unique=True) added_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), server_default=func.now()) diff --git a/app/rag/git_sources.py b/app/rag/git_sources.py index e2f7ea8..a350314 100644 --- a/app/rag/git_sources.py +++ b/app/rag/git_sources.py @@ -1,19 +1,26 @@ -"""The shared git-source resolver (phase 35, task 03). +"""The shared source resolver (phase 35, task 03; local kind, phase 38). One function, two callers: the in-app Sync pipeline (``app/api/sync.py::_run_sync``) and the CLI -(``scripts/import_docs.py::_resolve_sources``) — both resolve the repo -URLs to clone/pull through :func:`effective_git_sources`, so the -admin-managed ``git_sources`` table is what actually gets cloned and -indexed from either entry point (the API's ``GET /api/git-sources`` -fallback list is the only other place that reads the env list — task 02). +(``scripts/import_docs.py::_resolve_sources``) — both resolve the +``git_sources`` rows to import through :func:`effective_sources`, so +the admin-managed ``git_sources`` table is what actually gets cloned +(git rows) or walked directly (local rows) from either entry point +(the API's ``GET /api/git-sources`` fallback list is the only other +place that reads the env list — task 02). Precedence (the phase's locked decision — the env var is demoted, not -removed): ``git_sources`` DB rows win, in ``(added_at, id)`` order; -``BOR_GIT_SOURCES`` is a fallback only while the table is empty; both -empty → ``([], "env")`` and each caller keeps its existing fail-loud -behavior (sync: ``GitSyncError``; CLI: the legacy ``DEFAULT_SOURCES`` -fallback). +removed): ``git_sources`` DB rows of **both kinds** win, in +``(added_at, id)`` order; ``BOR_GIT_SOURCES`` is a fallback only while +the table is empty (git URLs surface as synthetic ``kind='git'`` rows — +the env fallback is git-only); both empty → ``([], "env")`` and each +caller keeps its existing fail-loud behavior (sync: ``GitSyncError``; +CLI: the legacy ``DEFAULT_SOURCES`` fallback). + +:func:`effective_git_sources` is kept as the phase-35 back-compat alias +(repo URLs of the effective git rows) so existing importers of the old +name keep working; new code calls :func:`effective_sources` and +branches on ``row.kind``. """ from __future__ import annotations @@ -26,19 +33,37 @@ from app.config import get_settings from app.models import GitSource -def effective_git_sources(db: Session) -> tuple[list[str], Literal["db", "env"]]: - """``(urls, origin)`` — the repo URLs to clone, and where they came from. +def effective_sources(db: Session) -> tuple[list[GitSource], Literal["db", "env"]]: + """``(rows, origin)`` — the effective source rows (both kinds) and + where they came from. - ``"db"``: the ``git_sources`` rows in ``(added_at, id)`` order (oldest - first, id tie-break for same-timestamp inserts) — once the table has - any row, ``BOR_GIT_SOURCES`` is ignored entirely. + ``"db"``: the ``git_sources`` rows — ``kind='git'`` (repo URL to + clone/pull) and ``kind='local'`` (existing directory to walk + directly) — in ``(added_at, id)`` order (oldest first, id tie-break + for same-timestamp inserts). Once the table has any row, + ``BOR_GIT_SOURCES`` is ignored entirely. ``"env"``: the ``BOR_GIT_SOURCES`` list — ``Settings.git_source_list``, - the phase-28 CSV parse, reused not re-implemented — used only while - the table is empty; both empty → ``([], "env")``. + the phase-28 CSV parse, reused not re-implemented — surfaced as + synthetic ``kind='git'`` rows (``path`` NULL; the env fallback is + git-only), used only while the table is empty; both empty → + ``([], "env")``. """ rows = db.scalars( select(GitSource).order_by(GitSource.added_at.asc(), GitSource.id.asc()) ).all() if rows: - return [row.url for row in rows], "db" - return list(get_settings().git_source_list), "env" + return list(rows), "db" + return [GitSource(url=url, kind="git") for url in get_settings().git_source_list], "env" + + +def effective_git_sources(db: Session) -> tuple[list[str], Literal["db", "env"]]: + """Back-compat alias for the phase-35 name (existing importers). + + Returns the repo URLs of the effective ``kind='git'`` rows only, with + the same origin label — the env fallback is git-only, so its + behavior is unchanged. Local rows carry no URL (they are walked + directly); new code should call :func:`effective_sources` and branch + on ``row.kind``. + """ + rows, origin = effective_sources(db) + return [row.url for row in rows if row.kind == "git"], origin diff --git a/app/schemas.py b/app/schemas.py index 84776a6..4b67210 100644 --- a/app/schemas.py +++ b/app/schemas.py @@ -181,28 +181,47 @@ class SteeringNoteList(BaseModel): class GitSourceIn(BaseModel): - """``POST /api/git-sources`` body: one repo URL (phase 35, task 02). + """``POST /api/git-sources`` body (phase 35, task 02; ``kind``, phase 38). - Mirrors :class:`SteeringNoteIn` — the URL is trimmed *before* the - length constraints run, so a whitespace-only body is a 422 and a URL - with surrounding spaces is stored clean. Shape validation - (``https://``, ``ssh://``, ``git@``) happens in the API layer so the - 422 detail can be one generic string that never echoes the input. + ``kind`` selects the source kind and which field carries its location: + + * ``"git"`` (default) — ``url`` is the repo URL. Mirrors the + phase-35 contract: trimmed *before* the length constraints run, so a + whitespace-only body is a 422 and a URL with surrounding spaces is + stored clean. Shape validation (``https://``, ``ssh://``, ``git@``) + and the kind-field rules (url present, no path) happen in the API + layer so the 422/409 details stay fixed strings that never echo the + input (credential safety). + * ``"local"`` — ``path`` is an existing directory on the server. + Trimmed here; the API layer then ``expanduser()``s it and requires an + absolute existing directory (else 422 naming the path — the path is + not a secret, unlike a git URL) and no ``url``. """ - url: str = Field(min_length=1, max_length=500) + kind: Literal["git", "local"] = "git" + url: str | None = Field(default=None, min_length=1, max_length=500) + path: str | None = Field(default=None, min_length=1, max_length=2000) @field_validator("url", mode="before") @classmethod def _trim_url(cls, v: object) -> object: return v.strip() if isinstance(v, str) else v + @field_validator("path", mode="before") + @classmethod + def _trim_path(cls, v: object) -> object: + return v.strip() if isinstance(v, str) else v + class GitSourceOut(BaseModel): - """One git source as returned by the API (phase 35, task 02). + """One created git source as returned by ``POST`` (phase 35, task 02). - ``id`` / ``added_at`` are nullable: env-fallback rows (table empty → - the list comes from ``BOR_GIT_SOURCES``) carry neither, only a URL. + ``id`` / ``added_at`` are non-null for a stored row. ``url`` is the + row's location column: the repo URL for ``kind=git`` rows and, for + ``kind=local`` rows, the stored (expanded) directory path — the + phase-35 response shape is unchanged by phase 38, so a local 201 + reports its path in ``url`` and the full row (``kind`` + ``path``) + via ``GET``. """ id: uuid.UUID | None @@ -210,14 +229,34 @@ class GitSourceOut(BaseModel): added_at: datetime | None +class GitSourceRow(BaseModel): + """One row of ``GET /api/git-sources`` (phase 35; ``kind``/``path``, + phase 38, task 02). + + ``kind`` discriminates the row: git rows (and the git-only + ``BOR_GIT_SOURCES`` env-fallback rows) carry ``url`` and + ``path: null``; local rows carry ``path`` (the absolute directory, + expanded) and the same string in ``url`` (the table's NOT-NULL + location column). ``id`` / ``added_at`` are nullable: env-fallback + rows (table empty) carry neither. + """ + + id: uuid.UUID | None + kind: Literal["git", "local"] + url: str + path: str | None + added_at: datetime | None + + class GitSourceList(BaseModel): """``GET /api/git-sources`` response (phase 35, task 02). ``from_env`` is True only when the ``git_sources`` table is empty and - the list comes from ``BOR_GIT_SOURCES`` (the phase's locked fallback); + the list comes from ``BOR_GIT_SOURCES`` (the phase's locked fallback + — env rows are git-only and report ``kind: "git"``, ``path: null``); once the table has rows the env var is ignored and ``from_env`` is False — the UI is the source of truth. """ - sources: list[GitSourceOut] + sources: list[GitSourceRow] from_env: bool diff --git a/frontend/assets/git-sources.js b/frontend/assets/git-sources.js index 4f770cc..bdf25aa 100644 --- a/frontend/assets/git-sources.js +++ b/frontend/assets/git-sources.js @@ -1,7 +1,10 @@ -/* Brain of Reese — Git sources admin page (phase 35, task 04). +/* Brain of Reese — Git sources admin page (phase 35, task 04; + * local directories, phase 38 task 04). * * The page module for /git-sources.html: the admin-only manager for the - * stored git source list (git-sources table, phase 35 tasks 01/02). + * stored source list (git-sources table, phase 35 tasks 01/02) — git + * repo URLs (kind "git") and existing local directories (kind + * "local", phase 38). * This module is the single owner of the page's behaviour: * * • boot — initSharedHeader() (one cached whoami, shared with the @@ -11,19 +14,25 @@ * #git-sources-content revealed, loadSources(). * • loadSources() — GET /api/git-sources → the table rows * (#git-sources-tbody), the env-fallback note's visibility - * (from_env), and the empty state. URLs are ALWAYS rendered with - * textContent — never innerHTML (they may embed user:pass@ + * (from_env), and the empty state. Each row leads with its kind + * badge (Git/Local — text + color, never color alone) plus the + * location in a mono : the git URL, or the full local path + * for kind "local" rows (phase 38). Values are ALWAYS rendered + * with textContent — never innerHTML (URLs may embed user:pass@ * credentials; phase 32's masking discipline). Non-2xx or a * network failure renders the role="alert" load error with a * retry button — never a stuck page. - * • add — #git-source-form submit → POST /api/git-sources {url}. - * §7.4 never-stale: the button disables + relabels "Adding…" - * while the request is out, re-enables ("Add source") on success - * AND failure. 201 clears the input, reloads the list, and - * focuses the new row's Remove button (a11y); a failure (409 - * duplicate, 422 shape) shows the server detail inline under the - * form (role="alert", 422 shape-aware like the tuning forms) and - * keeps the input — the instruction survives. + * • add — #git-source-form submit → POST /api/git-sources {url}; + * #local-source-form submit → POST /api/git-sources + * {kind: "local", path} (phase 38). ONE §7.4 never-stale + * lifecycle for both (wireAddForm): the button disables + + * relabels "Adding…" while the request is out, re-enables on + * success AND failure. 201 clears the input, reloads the list, + * and focuses the new row's Remove button (a11y); a failure (409 + * duplicate, 422 validation) shows the server detail inline under + * the form (role="alert", 422 shape-aware like the tuning forms) + * and keeps the input — the instruction survives. Local 422/409 + * details name the path (paths are not secrets, unlike URLs). * • remove — a row's Remove button asks window.confirm first * (removal prunes the documents only on the NEXT sync — the * confirm says so). Cancel → nothing; ok → the row button @@ -56,6 +65,12 @@ const formEl = document.querySelector("#git-source-form"); const urlInput = document.querySelector("#git-source-url"); const addBtn = document.querySelector("#git-source-add"); const addError = document.querySelector("#git-source-error"); +/* Phase 38: the second add form — "Local directory" (same element + contract as the git form, own ids). */ +const localFormEl = document.querySelector("#local-source-form"); +const pathInput = document.querySelector("#local-source-path"); +const localAddBtn = document.querySelector("#local-source-add"); +const localAddError = document.querySelector("#local-source-error"); const loadErrorEl = document.querySelector("#git-sources-load-error"); const loadErrorText = document.querySelector("#git-sources-load-error-text"); const retryBtn = document.querySelector("#git-sources-retry"); @@ -125,7 +140,7 @@ async function loadSources() { hideLoadError(); const sources = Array.isArray(data.sources) ? data.sources : []; renderSources(sources, data.from_env === true); - announce(`${sources.length} git source${sources.length === 1 ? "" : "s"} listed.`); + announce(`${sources.length} source${sources.length === 1 ? "" : "s"} listed.`); } function showLoadError(message) { @@ -153,11 +168,15 @@ function renderSources(sources, fromEnv) { if (emptyEl) emptyEl.hidden = hasRows; } -/* One row: the URL in a mono (textContent only — URLs may - contain credentials), the localized added date ("—" for env - fallback rows), and the per-row Remove button — or the "from .env" - tag for env-fallback rows (id null: nothing is stored to remove; - the env note says where the active list comes from). */ +/* One row: the kind badge (Git/Local — phase 38) followed by the + location in a mono (textContent only — git URLs may contain + credentials, local paths may contain anything), the localized added + date ("—" for env fallback rows), and the per-row Remove button — + or the "from .env" tag for env-fallback rows (id null: nothing is + stored to remove; the env note says where the active list comes + from). The value is the git URL for kind "git" rows and the full + local path for kind "local" rows (the API reports the path in both + `path` and `url`; `path` is the kind-typed field). */ const REMOVE_ICON = ''; @@ -165,12 +184,19 @@ function makeRow(s) { const tr = document.createElement("tr"); if (s.id) tr.dataset.id = s.id; + const isLocal = s.kind === "local"; + const value = isLocal ? (s.path ?? s.url) : s.url; + const kindLabel = isLocal ? "local" : "git"; + const urlTd = document.createElement("td"); urlTd.className = "git-source-url-cell"; - urlTd.title = s.url; // full URL on hover (long URLs scroll the wrapper) + urlTd.title = value; // full URL/path on hover (long values scroll the wrapper) + const badge = document.createElement("span"); + badge.className = `git-source-kind is-${isLocal ? "local" : "git"}`; + badge.textContent = isLocal ? "Local" : "Git"; // text + color, never color alone const code = document.createElement("code"); - code.textContent = s.url; // rendered as text, never as HTML - urlTd.appendChild(code); + code.textContent = value; // rendered as text, never as HTML + urlTd.append(badge, code); tr.appendChild(urlTd); const addedTd = document.createElement("td"); @@ -183,13 +209,13 @@ function makeRow(s) { const btn = document.createElement("button"); btn.type = "button"; btn.className = "git-source-remove"; - btn.setAttribute("aria-label", `Remove git source: ${s.url}`); + btn.setAttribute("aria-label", `Remove ${kindLabel} source: ${value}`); btn.innerHTML = REMOVE_ICON + "Remove"; const rowError = document.createElement("span"); rowError.className = "git-source-row-error"; rowError.setAttribute("role", "alert"); rowError.hidden = true; - btn.addEventListener("click", () => removeSource(s, btn, rowError)); + btn.addEventListener("click", () => removeSource(s, btn, rowError, kindLabel)); actTd.append(btn, rowError); } else { const tag = document.createElement("span"); @@ -206,9 +232,9 @@ function makeRow(s) { * (phase scope boundary), so the confirm says exactly that. Cancel → * nothing; a failed delete → per-row role="alert" error + re-enabled * button (never a stuck row); success → the list reloads. */ -async function removeSource(s, btn, rowError) { +async function removeSource(s, btn, rowError, kindLabel) { const ok = window.confirm( - "Remove this git source from the list? Its documents stay indexed until the next sync prunes them.", + `Remove this ${kindLabel} source from the list? Its documents stay indexed until the next sync prunes them.`, ); if (!ok) return; btn.disabled = true; // one delete per click @@ -216,43 +242,55 @@ async function removeSource(s, btn, rowError) { try { const r = await fetch(`/api/git-sources/${encodeURIComponent(s.id)}`, { method: "DELETE" }); if (!r.ok) { - rowError.textContent = await apiDetail(r, "Could not remove the git source — try again."); + rowError.textContent = await apiDetail(r, `Could not remove the ${kindLabel} source — try again.`); rowError.hidden = false; btn.disabled = false; return; } - announce("Git source removed."); + announce("Source removed."); await loadSources(); // 204: the server confirmed — the list re-renders } catch { - rowError.textContent = "Could not remove the git source — is the app reachable?"; + rowError.textContent = `Could not remove the ${kindLabel} source — is the app reachable?`; rowError.hidden = false; btn.disabled = false; } } -/* ---------- add (POST /api/git-sources) ---------- */ - -if (formEl && urlInput && addBtn) { - formEl.addEventListener("submit", async (e) => { +/* ---------- add (POST /api/git-sources) — both forms, one lifecycle + * (the local form is phase 38) ---------- + * The git form posts {url}; the local form posts {kind:"local",path}. + * wireAddForm gives both the §7.4 never-stale lifecycle: while the + * request is out the button disables + relabels "Adding…" and + * re-enables (idle label restored) on success AND failure. 201 clears + * the input, reloads the list, and focuses the new row's Remove button + * (a11y); a failure (409 duplicate, 422 validation) shows the server + * detail inline under the form (role="alert", 422 shape-aware via + * apiDetail) and keeps the input — the fix is one edit, not a re-type. + * Git 409/422 details are fixed generic strings (credential safety); + * local details name the path (not a secret). */ +function wireAddForm(opts) { + const { form, input, btn, error } = opts; + if (!form || !input || !btn) return; + form.addEventListener("submit", async (e) => { e.preventDefault(); // Client-side non-empty check (the input is `required` too — the // browser's native prompt is the first line, this one the second). - const url = urlInput.value.trim(); - if (!url) { - if (addError) { - addError.textContent = "Enter a git URL to add."; - addError.hidden = false; + const value = input.value.trim(); + if (!value) { + if (error) { + error.textContent = opts.emptyMessage; + error.hidden = false; } return; } - if (addError) addError.hidden = true; // a new attempt starts clean - addBtn.disabled = true; // §7.4: one POST per click - addBtn.textContent = "Adding…"; + if (error) error.hidden = true; // a new attempt starts clean + btn.disabled = true; // §7.4: one POST per click + btn.textContent = "Adding…"; try { const r = await fetch("/api/git-sources", { method: "POST", headers: { "Content-Type": "application/json" }, - body: JSON.stringify({ url }), + body: JSON.stringify(opts.body(value)), }); if (r.ok) { let createdId = null; @@ -261,31 +299,57 @@ if (formEl && urlInput && addBtn) { } catch { /* the 201 body is advisory — the reload is the truth */ } - urlInput.value = ""; // 201: the source is stored - announce("Git source added."); + input.value = ""; // 201: the source is stored + announce(opts.addedMessage); await loadSources(); // the new row lands in the table focusNewRow(createdId); // a11y: land the caret on the new row return; } - // 409 duplicate / 422 shape / anything else: the server detail - // inline (never echoing a URL the server wouldn't), form kept — - // the input survives so the fix is one edit, not a re-type. - if (addError) { - addError.textContent = await apiDetail(r, "Could not add the git source — try again."); - addError.hidden = false; + // 409 duplicate / 422 validation / anything else: the server + // detail inline, form kept — the input survives so the fix is + // one edit, not a re-type. + if (error) { + error.textContent = await apiDetail(r, opts.failMessage); + error.hidden = false; } } catch { - if (addError) { - addError.textContent = "Could not add the git source — is the app reachable?"; - addError.hidden = false; + if (error) { + error.textContent = opts.networkMessage; + error.hidden = false; } } finally { - addBtn.disabled = false; // never stale — success OR failure - addBtn.textContent = "Add source"; + btn.disabled = false; // never stale — success OR failure + btn.textContent = opts.idleLabel; } }); } +wireAddForm({ + form: formEl, + input: urlInput, + btn: addBtn, + error: addError, + body: (url) => ({ url }), + emptyMessage: "Enter a git URL to add.", + failMessage: "Could not add the git source — try again.", + networkMessage: "Could not add the git source — is the app reachable?", + addedMessage: "Git source added.", + idleLabel: "Add source", +}); + +wireAddForm({ + form: localFormEl, + input: pathInput, + btn: localAddBtn, + error: localAddError, + body: (path) => ({ kind: "local", path }), + emptyMessage: "Enter a directory path to add.", + failMessage: "Could not add the local directory — try again.", + networkMessage: "Could not add the local directory — is the app reachable?", + addedMessage: "Local source added.", + idleLabel: "Add directory", +}); + /* After a successful add, focus the new row's Remove button so the keyboard/screen-reader user lands where the new data is. The 201 body carries the row id (tr[data-id]); without one, the first row diff --git a/frontend/assets/styles.css b/frontend/assets/styles.css index bf408cd..4a0616b 100644 --- a/frontend/assets/styles.css +++ b/frontend/assets/styles.css @@ -1331,11 +1331,15 @@ details.thinking .thinking-text ul { margin: 0 0 0.5rem; } flex: 1; } -/* Add form — the tuning form's surface as a single row: visible label - + mono URL input (the credentials case is real, so the input is - mono) + the brand "Add source" button; wraps to a column at narrow - widths (the <=640px block below). */ -#git-source-form { +/* Add forms — the tuning form's surface as a single row: visible + label + mono location input (git URLs may embed credentials, local + paths may contain anything, so both inputs are mono) + the brand + button; wraps to a column at narrow widths (the <=640px block + below). Phase 38: the "Local directory" form (#local-source-form) + reuses the git form's rules VERBATIM — one form language for both + kinds. */ +#git-source-form, +#local-source-form { display: flex; flex-wrap: wrap; align-items: center; @@ -1346,9 +1350,12 @@ details.thinking .thinking-text ul { margin: 0 0 0.5rem; } box-shadow: var(--shadow); padding: 0.9rem 1rem 1rem; } -#git-source-form:focus-within { border-color: var(--brand); box-shadow: 0 0 0 3px var(--brand-soft), var(--shadow); } -#git-source-form > label { color: var(--ink); font-weight: 600; white-space: nowrap; } -#git-source-url { +#git-source-form:focus-within, +#local-source-form:focus-within { border-color: var(--brand); box-shadow: 0 0 0 3px var(--brand-soft), var(--shadow); } +#git-source-form > label, +#local-source-form > label { color: var(--ink); font-weight: 600; white-space: nowrap; } +#git-source-url, +#local-source-path { flex: 1; min-width: 14rem; min-height: 44px; @@ -1360,9 +1367,12 @@ details.thinking .thinking-text ul { margin: 0 0 0.5rem; } border-radius: var(--radius-sm); padding: 0.45rem 0.7rem; } -#git-source-url::placeholder { color: var(--ink-soft); } -#git-source-url:focus-visible { outline-offset: 0; border-color: var(--brand); } -#git-source-add { +#git-source-url::placeholder, +#local-source-path::placeholder { color: var(--ink-soft); } +#git-source-url:focus-visible, +#local-source-path:focus-visible { outline-offset: 0; border-color: var(--brand); } +#git-source-add, +#local-source-add { display: inline-flex; align-items: center; justify-content: center; @@ -1376,8 +1386,10 @@ details.thinking .thinking-text ul { margin: 0 0 0.5rem; } font-weight: 700; cursor: pointer; } -#git-source-add:hover:not(:disabled) { background: #7d88f5; } -#git-source-add:disabled { opacity: 0.6; cursor: wait; } +#git-source-add:hover:not(:disabled), +#local-source-add:hover:not(:disabled) { background: #7d88f5; } +#git-source-add:disabled, +#local-source-add:disabled { opacity: 0.6; cursor: wait; } /* The add form's inline error (role=alert): the err pair (9.1:1); flex-basis 100% drops it onto its own row under the input. */ @@ -1491,10 +1503,29 @@ details.thinking .thinking-text ul { margin: 0 0 0.5rem; } position: sticky; top: 0; } -/* URL cell: mono at the Sources-table size; the is plain - (no chip background — the cell IS the mono readout). */ +/* Location cell: mono at the Sources-table size; the is plain + (no chip background — the cell IS the mono readout). Phase 38: + leads with the kind badge (Git/Local) — the kinds are told apart by + TEXT as much as color (WCAG 1.4.1), and both badge pairs are AA: + brand-ink on brand-soft 6.9:1 (Git), ok-ink on ok-bg 10.6:1 + (Local). */ .git-sources-table td.git-source-url-cell { font-family: var(--mono); font-size: 0.82rem; } .git-sources-table td.git-source-url-cell code { font-family: inherit; } +.git-source-kind { + display: inline-block; + margin-right: 0.55rem; + padding: 0.08rem 0.5rem; + border-radius: 999px; + font-family: var(--font); + font-size: 0.72rem; + font-weight: 700; + letter-spacing: 0.04em; + text-transform: uppercase; + vertical-align: 0.06em; + white-space: nowrap; +} +.git-source-kind.is-git { color: var(--brand-ink); background: var(--brand-soft); } +.git-source-kind.is-local { color: var(--ok-ink); background: var(--ok-bg); } .git-sources-table tbody tr:hover { background: var(--bg); } .git-sources-table tbody tr:last-child td { border-bottom: 0; } @@ -2102,13 +2133,18 @@ details.thinking .thinking-text ul { margin: 0 0 0.5rem; } .doc-modal-meta { padding-inline: 0.9rem; } .doc-modal-content { padding: 0.75rem 0.9rem 1.25rem; } .composer { padding: 0.5rem; } - /* Phase 35: the git sources add form stacks like the other cards — - label, full-width mono input, full-width button; the table - wrapper's horizontal scroll already covers long URLs. */ - #git-source-form { flex-direction: column; align-items: stretch; } - #git-source-form > label { white-space: normal; } - #git-source-url { min-width: 0; } - #git-source-add { width: 100%; } + /* Phase 35 (phase 38: + the local directory form): the add forms + stack like the other cards — label, full-width mono input, + full-width button; the table wrapper's horizontal scroll already + covers long URLs/paths. */ + #git-source-form, + #local-source-form { flex-direction: column; align-items: stretch; } + #git-source-form > label, + #local-source-form > label { white-space: normal; } + #git-source-url, + #local-source-path { min-width: 0; } + #git-source-add, + #local-source-add { width: 100%; } .footer-inner { flex-direction: column; gap: 0.2rem; text-align: center; } main { padding-bottom: env(safe-area-inset-bottom, 0); } } diff --git a/frontend/git-sources.html b/frontend/git-sources.html index 81e294f..0db05b0 100644 --- a/frontend/git-sources.html +++ b/frontend/git-sources.html @@ -133,8 +133,9 @@

Git sources

- The repositories the Sync button clones and indexes. Add or - remove them here — no .env, no restart. + The git repositories and local directories the Sync button + imports. Add or remove them here — no .env, no + restart.

@@ -178,12 +179,36 @@ -
+ +
+ + + + +
+ +
- + - + @@ -195,16 +220,18 @@ - + + source does NOT clone or prune — the Sync button performs + that. The hint says so (phase 38: git + local together, + files removed from a source pruned). -->

- Use the Sync sources button in the header (or on - the Sources page) to clone the repos and refresh the index — - removing a repository prunes its documents from the index on the - next sync. + Sync clones/pulls the git repos and imports the local + directories together (files removed from a source are + pruned). Use the Sync sources button in the + header (or on the Sources page) to run it — removing a source + prunes its documents from the index on the next sync.

URLSource Added Actions