feat(admin): local directory sources — kind/path on git_sources, combined sync + import, page form + badges
An existing, non-git directory is now a first-class source alongside
the git repos: one table (git_sources + kind discriminator — A13
reversible migration), one admin page, one Sync button (phase locked
decisions; the phase-35 table is extended, not duplicated). The DB is
the local-source registry — no env var for local paths;
BOR_GIT_SOURCES stays a git-only empty-table fallback.
Migration 0007 (reversible, up/down integration-tested):
git_sources.kind TEXT NOT NULL DEFAULT 'git' + ck_git_sources_kind
(kind IN ('git','local')); git_sources.path TEXT NULL +
uq_git_sources_path (mirrors 0006's uq_git_sources_url). Existing rows
read kind='git', path=NULL.
API (phase-35 contract extended, git byte-identical): POST kind=local
requires path — trimmed, ~-expanded, absolute + an existing server
directory, else 422 naming the path (fail loud at add-time); duplicate
path 409 (named); wrong field combos 422. GET rows carry kind + path
(git and env rows: path null); anonymous still 403 on every route (A10).
Sync + import_docs resolve DB git + local rows together: git →
clone_or_pull (unchanged); local → re-verified .is_dir() AT SYNC TIME
(it may have moved/deleted since add-time) — a missing dir raises
"local source missing: <path>" (sanitized) before anything imports;
one import_sources(..., prune=True) over the single combined list
(pruning covers the union). Both-empty fails loudly ("no sources
configured (git or local)"); --source still wins; the env fallback
stays git-only.
Page: second "Add a local directory" form (the same §7.4 never-stale
button + inline-error lifecycle as the git form; 422/409 details name
the path), Git/Local badges on rows (text + color, never color alone —
WCAG), updated hint (git + local together, union prune); the
anonymous sign-in gate is unchanged.
Tests: 0007 up/down; the API local-kind matrix (403/201/422/409) with
the git-kind suite green unchanged; the sync pipeline local/git/
mixed/missing against a host temp dir (the KB actually updated);
import_docs DB resolution + --source precedence. Story E2E (isolated,
deterministic across runs): add (Local badge) → missing path inline
422 naming it / duplicate 409 → the real Sync button imports the
fixture file (GET /api/docs + sentinel in its content) → file deleted
+ sync prunes it (union prune) → row removed; anonymous gate + 403s
(phase-35 regression). test_git_sources_admin.py (phase 35) green
UNCHANGED — no selector collision with the new form;
test_sync_button.py green.
Docs: README — the two managed kinds (git = clone/pull mirror; local =
direct in-place walk), add-time validation, union pruning, "the DB is
the local-source registry (no env var for local paths)";
.env.example — the env fallback is git-only.
This commit is contained in:
@@ -0,0 +1,24 @@
|
||||
# Task 01 — `git_sources.kind` + `path` (migration 0007)
|
||||
|
||||
**Phase:** `38_local_directory_sources` · **Source:** `TODO.md:11 — "Also need a way to import from existing directory if it's not a git repo"`
|
||||
**Story:** `.agent/user_stories/local-directory-sources.md`
|
||||
|
||||
## Objective
|
||||
Extend the phase-35 `git_sources` table with a source-kind discriminator (`git` | `local`) and an optional local path, via a reversible migration — the foundation for the API, pipeline, and page tasks.
|
||||
|
||||
## Work
|
||||
1. `alembic/versions/0007_git_sources_kind.py` — read `alembic/versions/0006_git_sources.py` first and chain from its **actual** `revision` id (filenames are not revision ids — the phase-35 task-01 convention):
|
||||
- `upgrade()`: `ALTER TABLE git_sources ADD COLUMN kind TEXT NOT NULL DEFAULT 'git'` + `ADD CONSTRAINT ck_git_sources_kind CHECK (kind IN ('git', 'local'))`; `ALTER TABLE git_sources ADD COLUMN path TEXT`; a unique index on `path` — a partial unique index (`WHERE path IS NOT NULL`) where the 0006 style allows it, otherwise a plain unique index (Postgres treats NULLs as distinct, and the API enforces local-only paths anyway — mirror whatever 0006 chose for `url`).
|
||||
- `downgrade()`: drop the index, constraint, and columns in reverse.
|
||||
2. `app/models.py` — extend the phase-35 `GitSource` model: `kind: Mapped[str]` (default `"git"`) + `path: Mapped[str | None]` (nullable, unique) and update the docstring (phase 38: kind discriminator; local rows carry `path`, git rows keep `url`).
|
||||
3. Apply to the dev database: `podman compose up -d db` (if needed) then `uv run alembic upgrade head`; verify the columns + constraint + that existing rows read as `kind='git'`, `path=NULL`.
|
||||
4. Migration test — follow the 0004/0005/0006 pattern in `tests/integration/`: up adds the columns/constraint/index (existing rows keep `kind='git'`), down drops them (round-trip on the test DB).
|
||||
|
||||
## Testing & Quality
|
||||
- Integration: the up/down test above; full `uv run pytest` green.
|
||||
- Coverage: model + migration only — the `app/` gate stays >90%.
|
||||
|
||||
## Completion Criteria
|
||||
- [ ] `alembic/versions/0007_git_sources_kind.py` chains off 0006's real revision id; up/down reversible.
|
||||
- [ ] `uv run alembic upgrade head` applies cleanly; existing rows read `kind='git'`, `path=NULL`.
|
||||
- [ ] The 0007 up/down integration test passes; full `uv run pytest` green; `uv run ruff check . && uv run pyright` clean.
|
||||
@@ -0,0 +1,27 @@
|
||||
# Task 02 — The admin API: the local kind
|
||||
|
||||
**Phase:** `38_local_directory_sources` · **Source:** `TODO.md:11 — "Also need a way to import from existing directory if it's not a git repo"`
|
||||
**Story:** `.agent/user_stories/local-directory-sources.md`
|
||||
|
||||
## Objective
|
||||
Extend the phase-35 git-sources CRUD to accept and return `kind=local` rows with fail-loud path validation, keeping the git contract byte-identical.
|
||||
|
||||
## Work
|
||||
1. The phase-35 API module (the file behind `/api/git-sources` — identify it in the repo; phase 35 created it):
|
||||
- `POST /api/git-sources` body: `{kind?: "git"|"local" (default "git"), url?, path?}`:
|
||||
- `kind=git` → exactly today's `url` validation (trimmed, 1–500 chars, the `https?://` / `ssh://` / `git@` shape, 409 on duplicate without echoing the URL).
|
||||
- `kind=local` → `path` required: trimmed; `Path(p).expanduser()`; must be **absolute after expansion** and an **existing directory on the server** → else `422 {detail: "local source path is not a directory: <path>"}` (a missing path is a user error — fail loud at add-time so the owner sees it immediately; the path is not a secret, so echo it); 409 on a duplicate `path` (detail may name the path).
|
||||
- Wrong field combinations (git without url, local without path, both kinds' fields) → 422.
|
||||
- `GET /api/git-sources` rows gain `kind` + `path` (git rows: `path: null`; the env-fallback rows report `kind: "git"`; `from_env: true` semantics unchanged — env rows are git-only).
|
||||
- `DELETE /api/git-sources/{id}` — unchanged (removal prunes on the next sync, as today).
|
||||
2. The phase-35 request/response models (wherever they live — `app/schemas.py` or the module) — extend for the new fields; keep the OpenAPI docs accurate.
|
||||
3. Integration tests (extend `tests/integration/test_git_sources_api.py`): anonymous → 403 on all routes (regression); `kind=local` + an existing temp dir → 201 with the stored row (`kind=local`, `path` stored expanded); a relative path → 422; a missing path → 422 naming the path; a duplicate path → 409; a git row still validates exactly as before (regression); GET mixes kinds in added order.
|
||||
|
||||
## Testing & Quality
|
||||
- Integration: the matrix above; the existing git-kind suite green unchanged.
|
||||
- Coverage: **>90%** on the modified module.
|
||||
|
||||
## Completion Criteria
|
||||
- [ ] The local-kind POST matrix (201 / 422 / 409) green; the git contract unchanged.
|
||||
- [ ] GET rows carry `kind` + `path`; the env fallback stays git-only.
|
||||
- [ ] Full `uv run pytest` green; `uv run ruff check . && uv run pyright` clean.
|
||||
@@ -0,0 +1,30 @@
|
||||
# Task 03 — Sync + `import_docs`: git and local together
|
||||
|
||||
**Phase:** `38_local_directory_sources` · **Source:** `TODO.md:11 — "Also need a way to import from existing directory if it's not a git repo"`
|
||||
**Story:** `.agent/user_stories/local-directory-sources.md`
|
||||
|
||||
## Objective
|
||||
The canonical mirror action (the Sync button) and the CLI import DB **git + local** rows in one run: git rows clone/pull as today, local rows are walked directly; a missing local directory aborts the run loudly before anything is imported.
|
||||
|
||||
## Work
|
||||
1. The phase-35 resolution module (`app/rag/git_sources.py` or wherever `effective_git_sources()` lives — identify it in the repo) — extend:
|
||||
- `effective_sources(db) -> list[GitSource]` — the DB rows of **both** kinds (DB-wins / env-fallback semantics unchanged: while the table is empty, the env git URLs surface as synthetic `kind='git'` rows with `from_env`). Keep a backward-compatible alias (`effective_git_sources`) if other modules import the old name.
|
||||
- Log the list origin as today (`origin=db|env`) plus the kind counts (`git=N local=M`).
|
||||
2. `app/api/sync.py::_run_sync`:
|
||||
- For each resolved row: `kind=git` → `clone_or_pull(url, sources_root / repo_name(url))` (unchanged); `kind=local` → `Path(row.path).expanduser()`, verify `.is_dir()` **at sync time** (the directory may have moved/deleted since add-time) → else raise a sync error `local source missing: <path>` (no credentials involved, but run it through the existing `_sanitize_error` for consistency).
|
||||
- `import_sources(combined, llm, prune=True)` over the **single combined list** (git checkouts + local dirs) — pruning covers the union (phase-32 semantics); the overview regeneration is unchanged (change-gated).
|
||||
- The fail-loud empty-config check: no git rows, no local rows, and no env URLs → `"no sources configured (git or local)"` (replaces phase 32's git-only message).
|
||||
3. `scripts/import_docs.py` — `_resolve_sources` gains the DB path (after `--source`, which still wins over everything): no `--source` and the DB table non-empty → the combined list (git cloned/pulled + local direct); table empty → env git URLs (as today) → the legacy `DEFAULT_SOURCES`. A missing local dir → abort with the path named, before importing anything (the same pre-import fail-loud as a failing git clone).
|
||||
4. Unit/integration:
|
||||
- Unit: `effective_sources` — mixed kinds, DB-wins, env fallback git-only, both-empty (extend the phase-35 tests).
|
||||
- Integration (the `test_sync_api.py` pattern, with a **temp local dir** — a host temp dir containing one fixture `.md`, since the app server runs on the same host): local-only, git-only, and mixed syncs — the local file lands in the KB (`GET /api/docs` as admin); a missing local path → status `failed` with `local source missing: …` in the sanitized error; `import_docs` (no `--source`) with a DB local row imports it; `--source` still wins over the DB.
|
||||
|
||||
## Testing & Quality
|
||||
- The suites above; `test_git_sources_api.py`, `test_sync_api.py`, `test_import_docs_git.py` (phase 35's list) stay green through the indirection.
|
||||
- Coverage: **>90%** on the modified modules.
|
||||
|
||||
## Completion Criteria
|
||||
- [ ] A mixed git + local sync imports both in one run; union pruning works (a file deleted from the local dir is pruned on the next sync).
|
||||
- [ ] A missing local dir fails the run loudly (the status error names the path) and imports nothing.
|
||||
- [ ] `import_docs`: DB git + local resolution; `--source` wins; the env fallback is git-only; the both-empty message is updated.
|
||||
- [ ] Full `uv run pytest` green; coverage gate holds; `uv run ruff check . && uv run pyright` clean.
|
||||
@@ -0,0 +1,24 @@
|
||||
# Task 04 — The page: the local-directory form + badges
|
||||
|
||||
**Phase:** `38_local_directory_sources` · **Source:** `TODO.md:11 — "Also need a way to import from existing directory if it's not a git repo"`
|
||||
**Story:** `.agent/user_stories/local-directory-sources.md`
|
||||
|
||||
## Objective
|
||||
The sources page (phase 35) can add a **local directory** next to git repos, the list shows which is which, and the hint reflects the combined Sync semantics.
|
||||
|
||||
## Work
|
||||
1. The phase-35 page (the git-sources HTML template + its page script — identify the actual paths in the repo):
|
||||
- A second add form, **"Local directory"**: a labeled text input (placeholder `~/Notes`), an Add button, an inline error slot — the same never-stale-button + inline-error pattern as the git form (PLAN §7.4); on success the form clears and the list re-fetches.
|
||||
- List rows: a kind badge — `Git` / `Local` (a small styled span, distinguishable by color **and** text, not color alone — WCAG) — plus the mono value (git URL as today; the full local path) + added date + Remove (Remove is unchanged — it prunes on the next sync).
|
||||
- The hint text: "Sync clones/pulls the git repos and imports the local directories together (files removed from a source are pruned)."
|
||||
- Anonymous: the sign-in gate unchanged; the admin-only nav link unchanged (phase 35/29).
|
||||
2. `frontend/assets/styles.css` — the badge styles + the second form (reuse the existing form styles; contrast ≥ 4.5:1 in both themes).
|
||||
3. UI Structure Check (AGENTS.md rule 5) before finalizing: landmarks intact, both inputs labeled, focus-visible on the new Add button, the page stays inside the shared layout (no new top-level structure).
|
||||
|
||||
## Testing & Quality
|
||||
- Gated by the story E2E (task 05); no CDN (AGENTS.md rule 6 — no new external tags).
|
||||
|
||||
## Completion Criteria
|
||||
- [ ] The admin adds a local directory through the page; the row appears with the Local badge; an invalid path shows the 422 detail inline and the button recovers (never stale).
|
||||
- [ ] The git form + all existing page behavior unchanged.
|
||||
- [ ] `uv run pytest` green (no frontend unit layer — the E2E is the gate); `uv run ruff check . && uv run pyright` clean.
|
||||
+8
-2
@@ -55,8 +55,14 @@ BOR_AGENT_READ_CALLS=1 # per-turn read_document opportunities (0 disa
|
||||
# variable is the EMPTY-TABLE FALLBACK: it only applies while the admin
|
||||
# list is empty; once the page has stored any source, this variable is
|
||||
# ignored (the page is the source of truth). Empty table + empty variable
|
||||
# = no git sources (import_docs falls back to --source / the old
|
||||
# ~/Homelab + ~/Deployments defaults; the UI Sync button fails loudly).
|
||||
# + no local rows = no sources (import_docs falls back to --source / the
|
||||
# old ~/Homelab + ~/Deployments defaults; the UI Sync button fails loudly
|
||||
# with "no sources configured (git or local)").
|
||||
#
|
||||
# Phase 38: this env fallback is GIT-ONLY. Local directory sources (an
|
||||
# existing, non-git directory on the server) have NO env var — the DB is
|
||||
# the local-source registry: add them on the same admin page (the
|
||||
# "Add a local directory" form, kind 'local' + path, migration 0007).
|
||||
# BOR_GIT_SOURCES=https://github.com/user/homelab.git,https://github.com/user/deployments.git
|
||||
# BOR_SOURCES_DIR=~/bor-sources
|
||||
|
||||
|
||||
@@ -63,23 +63,29 @@ uv run python -m scripts.llm_probe # sanity: models + 768-dim check
|
||||
uv run python -m scripts.import_docs # import the configured sources (below)
|
||||
```
|
||||
|
||||
Two source modes:
|
||||
Two managed source kinds (one page, one registry) plus a manual override:
|
||||
|
||||
- **Git sources (recommended)** — the list is managed on the **admin
|
||||
Git sources page** (`/git-sources.html`) and stored in Postgres (see
|
||||
- **Git sources** — managed on the **admin Git sources page**
|
||||
(`/git-sources.html`) and stored in Postgres (see
|
||||
[Git-based sources](#git-based-sources)). `import_docs` clones each repo
|
||||
(first run) or pulls it (subsequent runs) into
|
||||
`BOR_SOURCES_DIR/<repo-name>/` (default `~/bor-sources`) and indexes the
|
||||
checkouts. While the stored list is empty, the `BOR_GIT_SOURCES` variable
|
||||
in `.env` is the fallback — the moment the page stores a source, the
|
||||
variable is ignored.
|
||||
- **Local directory sources** (phase 38) — an existing, non-git directory
|
||||
on the server, registered on the *same* admin page (see
|
||||
[Local directory sources](#local-directory-sources)). No clone, no
|
||||
checkout copy: the directory is walked in place. There is **no env var
|
||||
for local paths** — the DB is the registry.
|
||||
- **Manual directories** — `--source <path>` (repeatable) imports local
|
||||
directories directly and *always wins* over the git sources (stored list
|
||||
or env).
|
||||
directories directly and *always wins* over the stored sources (git and
|
||||
local) and the env fallback.
|
||||
- If neither is set (stored list, `--source`, and `BOR_GIT_SOURCES` all
|
||||
empty), the import falls back to the **previous** default,
|
||||
empty), `import_docs` falls back to the **previous** default,
|
||||
`~/Homelab` + `~/Deployments` — kept only for backwards compatibility,
|
||||
now replaced by the git sources list.
|
||||
now replaced by the managed sources; the UI Sync button instead fails
|
||||
loudly ("no sources configured (git or local)").
|
||||
|
||||
### 6. Run the app
|
||||
```bash
|
||||
@@ -107,12 +113,18 @@ uv run uvicorn app.main:app --reload
|
||||
new tab). **Admin-only** — anonymous visitors see a sign-in gate instead
|
||||
(the catalog is what the login locks; the document viewer itself stays
|
||||
open to everyone).
|
||||
- **Git sources** (`/git-sources.html`) — the list of git repositories the
|
||||
**Sync sources** button clones and indexes; **admin-only** (the same
|
||||
sign-in gate as Sources). Add or remove repositories here — no `.env`
|
||||
editing, no restart. Adding/removing does not clone or prune on its
|
||||
own: the Sync button performs that, and a removed repository's documents
|
||||
leave the index on the next sync.
|
||||
- **Git sources** (`/git-sources.html`) — the admin-managed source
|
||||
registry: the git repositories the **Sync sources** button clones and
|
||||
indexes, **and** existing local directories it imports directly
|
||||
(phase 38 — one table with a `kind` discriminator, one page); **admin-only**
|
||||
(the same sign-in gate as Sources). Add or remove sources here — no
|
||||
`.env` editing, no restart. A local directory must be an absolute,
|
||||
existing directory at add-time (a missing/relative path is rejected
|
||||
inline, naming the path; so are duplicates); list rows carry a **Git**
|
||||
or **Local** badge. Adding/removing does not clone or prune on its
|
||||
own: the Sync button performs that (git + local together, one run,
|
||||
prune over the union), and a removed source's documents leave the index
|
||||
on the next sync.
|
||||
|
||||
## Thinking
|
||||
|
||||
@@ -345,16 +357,52 @@ BOR_SOURCES_DIR=~/bor-sources # default; each repo lands in <dir>/<repo-name>/
|
||||
**nothing** (no partial junk). Fix the URL/connectivity and re-run — the
|
||||
other checkouts stay on disk and are pulled as usual.
|
||||
|
||||
### Local directory sources
|
||||
|
||||
Not every set of notes lives in a git repo — a plain directory can be a
|
||||
first-class source too (phase 38). It shares the git sources' **one
|
||||
table** (the `git_sources` registry with a `kind` discriminator: `git` |
|
||||
`local`, migration 0007), **one admin page**, and **one Sync button**:
|
||||
|
||||
- **Add it on the Git sources page** — the “Add a local directory” form
|
||||
next to the git form. Add-time validation fails loud: the path is
|
||||
trimmed, `~` is expanded, and must be an **absolute, existing directory
|
||||
on the server** — anything else (missing, relative, a file) is rejected
|
||||
with the path named inline; a duplicate path is rejected the same way.
|
||||
There is **no env var for local paths** — the DB is the local-source
|
||||
registry (`BOR_GIT_SOURCES` stays a git-only fallback).
|
||||
- **Sync walks it directly** — no clone, no checkout copy: each run
|
||||
indexes the directory in place (A9 format filter, hidden-dir skip,
|
||||
sha256 delta), together with the git checkouts in the **same run**.
|
||||
`documents.source` is the directory's name. The directory is
|
||||
re-verified to exist **at sync time** (it may have moved or been
|
||||
deleted since add-time): a missing directory fails the run loudly,
|
||||
naming the path, and imports **nothing** (the same pre-import fail-loud
|
||||
as a failing git clone).
|
||||
- **Pruning is over the union** — git checkouts and local directories are
|
||||
imported together with `prune=True`, so a file removed from a local
|
||||
directory, a repo, or a removed source leaves the index on that run.
|
||||
Removing the row on the page stops the directory being a source; its
|
||||
documents leave the index on the next sync (exactly like git sources).
|
||||
- **`import_docs`** (no `--source`) resolves the stored git **and** local
|
||||
rows — git cloned/pulled as above, local walked directly — in one run;
|
||||
`--source` still wins over everything; while the table is empty,
|
||||
`BOR_GIT_SOURCES` is the git-only fallback; no git rows, no local rows,
|
||||
and no env URLs fails loudly ("no sources configured (git or local)").
|
||||
|
||||
### Sync from the UI
|
||||
|
||||
The **Sync sources** button on the **Sources** page — visible to the
|
||||
**admin only** (anonymous visitors never see it) — runs the whole
|
||||
git-source refresh in one click, in-process:
|
||||
|
||||
1. **clone/pull** every configured git source — the admin-managed list
|
||||
(the `git_sources` table; `BOR_GIT_SOURCES` only while that list is
|
||||
empty), through the same `clone_or_pull` the CLI uses (shallow clone
|
||||
on first run, `git pull --ff-only` afterwards);
|
||||
1. **clone/pull + walk** every configured source — the git sources (the
|
||||
admin-managed `git_sources` table; `BOR_GIT_SOURCES` only while that
|
||||
list is empty) through the same `clone_or_pull` the CLI uses (shallow
|
||||
clone on first run, `git pull --ff-only` afterwards), **and** the
|
||||
local directories registered on the same page, walked directly
|
||||
(re-verified to exist at sync time — a missing directory fails the run
|
||||
loudly, naming the path);
|
||||
2. **re-import with prune** — the `--prune` equivalent, so files deleted
|
||||
upstream leave the index (the button is the canonical "mirror the
|
||||
repos" action); the sha256 delta still skips unchanged files, so an
|
||||
@@ -363,12 +411,12 @@ git-source refresh in one click, in-process:
|
||||
chat turn injects) — but only when the import actually changed the
|
||||
knowledge base.
|
||||
|
||||
- **Prerequisites:** at least one git source must be configured — a row
|
||||
on the admin Git sources page, or `BOR_GIT_SOURCES` in `.env` while the
|
||||
stored list is empty; **both** empty fails the sync loudly ("no git
|
||||
sources configured"), because the button targets the git repos only
|
||||
(manual `--source` directories have no repo to clone) — and `git` must
|
||||
be on the app's `PATH`.
|
||||
- **Prerequisites:** at least one source must be configured — a git or
|
||||
local row on the admin Git sources page, or `BOR_GIT_SOURCES` in `.env`
|
||||
while the stored list is empty (git-only); **all** empty fails the sync
|
||||
loudly ("no sources configured (git or local)"), because the button
|
||||
targets the admin-managed registry (manual `--source` directories have
|
||||
no place in it) — and `git` must be on the app's `PATH` for git sources.
|
||||
- **States:** clicking starts the run (`202`) and the button goes
|
||||
disabled with **Syncing…** (spinning icon) while the page polls
|
||||
`GET /api/sync/status` every 2 s. There is deliberately **no
|
||||
@@ -597,7 +645,7 @@ served locally (no CDN), `BOR_ENVIRONMENT=production`.
|
||||
| `BOR_AGENT_LIST_CALLS` | `1` | per-turn `list_documents` tool opportunities on grounded turns (0 disables the tool) |
|
||||
| `BOR_AGENT_READ_CALLS` | `1` | per-turn `read_document` tool opportunities on grounded turns (0 disables the tool) |
|
||||
| `BOR_IMPORT_EXTENSIONS` | `md,markdown,txt,yaml,yml,json,py` | csv of importable formats (may only narrow the A9 set) |
|
||||
| `BOR_GIT_SOURCES` | — (empty) | csv of git repo URLs — **fallback while the admin Git sources page's list (Postgres `git_sources`) is empty**; the page is the primary management surface (see *Git-based sources*) |
|
||||
| `BOR_GIT_SOURCES` | — (empty) | csv of git repo URLs — **fallback while the admin Git sources page's list (Postgres `git_sources`) is empty**; the page is the primary management surface (see *Git-based sources*). **Git-only**: local directory sources have no env var — they are registered on the admin page (see *Local directory sources*) |
|
||||
| `BOR_SOURCES_DIR` | `~/bor-sources` | where the git source repos are cloned/pulled (one subdirectory per repo) |
|
||||
| `BOR_STEERING_MAX_CHARS` | `8000` | char budget for the `<tuning>` (steering notes) prompt section |
|
||||
| `BOR_SUMMARY_MAX_CHARS` | `12000` | cap on document content sent to the `lite` summary model at import (see *Document summaries*) |
|
||||
|
||||
@@ -0,0 +1,49 @@
|
||||
"""git_sources.kind + path: local-directory source discriminator
|
||||
|
||||
Revision ID: 0007
|
||||
Revises: 0006
|
||||
Create Date: 2026-08-26
|
||||
|
||||
Phase 38 (local-directory-sources story, A13 — additive, reversible
|
||||
columns; the phase-35 table is extended, not duplicated):
|
||||
|
||||
* ``git_sources.kind`` — TEXT NOT NULL with server default ``'git'`` and
|
||||
check constraint ``ck_git_sources_kind`` (``kind IN ('git', 'local')``).
|
||||
Existing rows read as ``kind='git'``; git rows keep ``url``, local
|
||||
rows carry an existing directory under ``path`` (walked directly —
|
||||
no clone/pull).
|
||||
* ``git_sources.path`` — TEXT, nullable (NULL for git rows). Made unique
|
||||
by the plain unique index ``uq_git_sources_path`` — mirrors 0006's
|
||||
``uq_git_sources_url`` choice for ``url`` (Postgres treats NULLs as
|
||||
distinct under a unique index, and the API enforces local-only paths
|
||||
anyway).
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import sqlalchemy as sa
|
||||
|
||||
from alembic import op
|
||||
|
||||
revision = "0007"
|
||||
down_revision = "0006"
|
||||
branch_labels = None
|
||||
depends_on = None
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
op.add_column(
|
||||
"git_sources",
|
||||
sa.Column("kind", sa.Text(), server_default=sa.text("'git'"), nullable=False),
|
||||
)
|
||||
op.create_check_constraint(
|
||||
"ck_git_sources_kind", "git_sources", "kind IN ('git', 'local')"
|
||||
)
|
||||
op.add_column("git_sources", sa.Column("path", sa.Text(), nullable=True))
|
||||
op.create_index("uq_git_sources_path", "git_sources", ["path"], unique=True)
|
||||
|
||||
|
||||
def downgrade() -> None:
|
||||
op.drop_index("uq_git_sources_path", table_name="git_sources")
|
||||
op.drop_constraint("ck_git_sources_kind", table_name="git_sources", type_="check")
|
||||
op.drop_column("git_sources", "path")
|
||||
op.drop_column("git_sources", "kind")
|
||||
+117
-40
@@ -1,33 +1,42 @@
|
||||
"""Admin-managed git sources API (phase 35, task 02).
|
||||
"""Admin-managed sources API (phase 35, task 02; local kind, phase 38).
|
||||
|
||||
Admin-only CRUD under ``/api/git-sources`` (phase 16 pattern, A10
|
||||
extended — the public API surface stays stateless and the signed cookie
|
||||
remains the only session state, same as ``/api/steering`` and
|
||||
``/api/sync``): the ``git_sources`` table holds the repo URLs the Sync
|
||||
button (phase 32) and ``import_docs`` (phase 28) clone/pull. DB rows win
|
||||
over ``BOR_GIT_SOURCES``, which is a fallback while the table is empty
|
||||
(the phase's locked decision — ``from_env`` tells the UI which list it is
|
||||
looking at, so the page can show the env note only while the fallback is
|
||||
active).
|
||||
``/api/sync``): the ``git_sources`` table holds the sources the Sync
|
||||
button (phase 32) and ``import_docs`` (phase 28) import — ``kind='git'``
|
||||
rows carry the repo URL to clone/pull, ``kind='local'`` rows (phase 38)
|
||||
carry an existing directory on the server to walk directly. DB rows win
|
||||
over ``BOR_GIT_SOURCES``, which is a git-only fallback while the table is
|
||||
empty (the phase's locked decision — ``from_env`` tells the UI which list
|
||||
it is looking at, so the page can show the env note only while the
|
||||
fallback is active).
|
||||
|
||||
Routes: ``GET`` (DB rows oldest-first, or the env list with
|
||||
``from_env: true`` while the table is empty), ``POST`` (201, validated
|
||||
create), ``DELETE /{source_id}`` (204). The whole router sits behind
|
||||
:func:`app.core.auth.require_admin` — anonymous callers get 403 on every
|
||||
route.
|
||||
``from_env: true`` while the table is empty; rows carry ``kind`` +
|
||||
``path``, git rows — and env rows — report ``path: null``), ``POST``
|
||||
(201, validated create; ``kind`` selects the validation: git → exactly
|
||||
the phase-35 URL contract, local → an existing absolute directory, else
|
||||
422 naming the path), ``DELETE /{source_id}`` (204). The whole router
|
||||
sits behind :func:`app.core.auth.require_admin` — anonymous callers get
|
||||
403 on every route.
|
||||
|
||||
No credential-echo path: URLs may embed ``user:pass@`` (phase 32's
|
||||
masking discipline), so the 409/422 details are fixed generic strings
|
||||
that never repeat the submitted URL.
|
||||
No credential-echo path: git URLs may embed ``user:pass@`` (phase 32's
|
||||
masking discipline), so every git 409/422 detail is a fixed generic
|
||||
string that never repeats the submitted URL. Local paths are not
|
||||
secrets — the local 422/409 details name the (expanded) path so the
|
||||
owner sees exactly which directory failed.
|
||||
|
||||
Scope boundary (phase locked decisions): adding or removing a repo does
|
||||
NOT clone, import, or prune anything — the existing Sync button performs
|
||||
that (a removal prunes on the next sync, ``prune=True``).
|
||||
Scope boundary (phase locked decisions): adding or removing a source
|
||||
does NOT clone, import, or prune anything — the existing Sync button
|
||||
performs that (a removal prunes on the next sync, ``prune=True``).
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import re
|
||||
import uuid
|
||||
from pathlib import Path
|
||||
from typing import Literal, cast
|
||||
|
||||
from fastapi import APIRouter, Depends, HTTPException, Response
|
||||
from sqlalchemy import select
|
||||
@@ -38,7 +47,7 @@ from app.config import get_settings
|
||||
from app.core.auth import require_admin
|
||||
from app.db import get_db
|
||||
from app.models import GitSource
|
||||
from app.schemas import GitSourceIn, GitSourceList, GitSourceOut
|
||||
from app.schemas import GitSourceIn, GitSourceList, GitSourceOut, GitSourceRow
|
||||
|
||||
router = APIRouter(
|
||||
prefix="/git-sources",
|
||||
@@ -57,24 +66,37 @@ URL_RE = re.compile(r"^(https?://|ssh://|git@)")
|
||||
def list_git_sources(
|
||||
db: Session = Depends(get_db), # noqa: B008
|
||||
) -> GitSourceList:
|
||||
"""The effective git source list.
|
||||
"""The effective source list (git + local rows, phase 38).
|
||||
|
||||
DB rows ordered by ``(added_at, id)`` (oldest first, id tie-break for
|
||||
same-timestamp inserts) with ``from_env: false``; while the table is
|
||||
empty, the ``BOR_GIT_SOURCES`` env URLs as rows with null
|
||||
``id``/``added_at`` and ``from_env: true``.
|
||||
same-timestamp inserts) with ``from_env: false`` — each row carries
|
||||
its ``kind`` and, for local rows, the stored ``path`` (git rows and
|
||||
env rows report ``path: null``); while the table is empty, the
|
||||
``BOR_GIT_SOURCES`` env URLs as git rows (the env fallback is
|
||||
git-only) with null ``id``/``added_at`` and ``from_env: true``.
|
||||
"""
|
||||
rows = db.scalars(
|
||||
select(GitSource).order_by(GitSource.added_at.asc(), GitSource.id.asc())
|
||||
).all()
|
||||
if rows:
|
||||
return GitSourceList(
|
||||
sources=[GitSourceOut(id=row.id, url=row.url, added_at=row.added_at) for row in rows],
|
||||
sources=[
|
||||
# ``ck_git_sources_kind`` (migration 0007) guarantees the
|
||||
# value is 'git' or 'local' — the cast documents that.
|
||||
GitSourceRow(
|
||||
id=row.id,
|
||||
kind=cast(Literal["git", "local"], row.kind),
|
||||
url=row.url,
|
||||
path=row.path,
|
||||
added_at=row.added_at,
|
||||
)
|
||||
for row in rows
|
||||
],
|
||||
from_env=False,
|
||||
)
|
||||
return GitSourceList(
|
||||
sources=[
|
||||
GitSourceOut(id=None, url=url, added_at=None)
|
||||
GitSourceRow(id=None, kind="git", url=url, path=None, added_at=None)
|
||||
for url in get_settings().git_source_list
|
||||
],
|
||||
from_env=True,
|
||||
@@ -86,14 +108,49 @@ def create_git_source(
|
||||
payload: GitSourceIn,
|
||||
db: Session = Depends(get_db), # noqa: B008
|
||||
) -> GitSourceOut:
|
||||
"""Store one repo URL (already trimmed by the schema).
|
||||
"""Store one source (fields already trimmed by the schema).
|
||||
|
||||
422 when the shape is not one of the accepted prefixes (generic
|
||||
detail — the input is never echoed); 409 when the trimmed URL is
|
||||
already stored (same; the unique index is the backstop against a
|
||||
concurrent insert the pre-check missed); 201 + the created row
|
||||
otherwise.
|
||||
``kind="git"`` (default) — exactly the phase-35 contract: 422 when
|
||||
the URL shape is not one of the accepted prefixes (generic detail —
|
||||
the input is never echoed), 409 when the trimmed URL is already
|
||||
stored (the unique index is the backstop against a concurrent insert
|
||||
the pre-check missed), 201 + the created row otherwise.
|
||||
|
||||
``kind="local"`` — ``path`` must expand (``~``) to an absolute,
|
||||
existing directory on the server: 422 naming the path otherwise
|
||||
(fail loud at add-time — the owner sees it immediately), 409 when
|
||||
the path is already stored (detail names the path), 201 + the stored
|
||||
row otherwise (``url`` holds the expanded path — the table's
|
||||
NOT-NULL location column).
|
||||
|
||||
Wrong field combinations (git without url, local without path, both
|
||||
kinds' fields) are 422 with fixed, input-free details.
|
||||
"""
|
||||
row = _create_git_row(payload, db) if payload.kind == "git" else _create_local_row(payload, db)
|
||||
return GitSourceOut(id=row.id, url=row.url, added_at=row.added_at)
|
||||
|
||||
|
||||
def _commit_new(row: GitSource, duplicate_detail: str, db: Session) -> GitSource:
|
||||
"""Insert ``row``; the unique index is the backstop — a concurrent
|
||||
insert the pre-check missed still yields the generic 409, never a
|
||||
500 (phase-35 convention, now shared by both kinds)."""
|
||||
db.add(row)
|
||||
try:
|
||||
db.commit()
|
||||
except IntegrityError:
|
||||
db.rollback()
|
||||
raise HTTPException(status_code=409, detail=duplicate_detail) from None
|
||||
db.refresh(row)
|
||||
return row
|
||||
|
||||
|
||||
def _create_git_row(payload: GitSourceIn, db: Session) -> GitSource:
|
||||
"""``kind=git`` — the phase-35 URL contract, unchanged (A10: no
|
||||
credential echo, so every detail is a fixed string)."""
|
||||
if payload.path is not None:
|
||||
raise HTTPException(status_code=422, detail="a git source takes a url, not a path")
|
||||
if payload.url is None:
|
||||
raise HTTPException(status_code=422, detail="a git source requires a url")
|
||||
url = payload.url
|
||||
if not URL_RE.match(url):
|
||||
raise HTTPException(
|
||||
@@ -101,17 +158,37 @@ def create_git_source(
|
||||
)
|
||||
if db.scalar(select(GitSource).where(GitSource.url == url)) is not None:
|
||||
raise HTTPException(status_code=409, detail="a git source with this URL already exists")
|
||||
row = GitSource(url=url)
|
||||
db.add(row)
|
||||
try:
|
||||
db.commit()
|
||||
except IntegrityError:
|
||||
db.rollback()
|
||||
return _commit_new(
|
||||
GitSource(url=url, kind="git"), "a git source with this URL already exists", db
|
||||
)
|
||||
|
||||
|
||||
def _create_local_row(payload: GitSourceIn, db: Session) -> GitSource:
|
||||
"""``kind=local`` — fail-loud add-time validation (phase 38):
|
||||
trimmed → ``expanduser()`` → absolute + existing directory, else 422
|
||||
naming the path (not a secret, unlike a git URL)."""
|
||||
if payload.url is not None:
|
||||
raise HTTPException(status_code=422, detail="a local source takes a path, not a url")
|
||||
if payload.path is None:
|
||||
raise HTTPException(status_code=422, detail="a local source requires a path")
|
||||
expanded = Path(payload.path).expanduser()
|
||||
if not expanded.is_absolute() or not expanded.is_dir():
|
||||
raise HTTPException(
|
||||
status_code=409, detail="a git source with this URL already exists"
|
||||
) from None
|
||||
db.refresh(row)
|
||||
return GitSourceOut(id=row.id, url=row.url, added_at=row.added_at)
|
||||
status_code=422, detail=f"local source path is not a directory: {expanded}"
|
||||
)
|
||||
path = str(expanded)
|
||||
if db.scalar(select(GitSource).where(GitSource.path == path)) is not None:
|
||||
raise HTTPException(
|
||||
status_code=409, detail=f"a local source with this path already exists: {path}"
|
||||
)
|
||||
# ``url`` is the table's NOT-NULL location column (phase 38: local
|
||||
# rows carry the expanded path there too — git URL shapes and absolute
|
||||
# paths cannot collide).
|
||||
return _commit_new(
|
||||
GitSource(url=path, kind="local", path=path),
|
||||
f"a local source with this path already exists: {path}",
|
||||
db,
|
||||
)
|
||||
|
||||
|
||||
@router.delete("/{source_id}", status_code=204)
|
||||
|
||||
+49
-25
@@ -12,20 +12,27 @@ plus a module-level :class:`SyncStatus` that the UI polls every 2 s
|
||||
409; the status object is authoritative, so the UI can never sit on a
|
||||
stale button state (§7.4 adaptation, phase locked decisions).
|
||||
|
||||
Pipeline (the canonical "mirror the repos" action — phase locked
|
||||
Pipeline (the canonical "mirror the sources" action — phase locked
|
||||
decisions):
|
||||
|
||||
1. resolve the effective git sources — the ``git_sources`` DB rows,
|
||||
else the ``BOR_GIT_SOURCES`` fallback
|
||||
(:func:`app.rag.git_sources.effective_git_sources`, shared with the
|
||||
CLI) — empty on both origins fails loudly (``no git sources
|
||||
configured``) instead of silently importing the legacy local
|
||||
directories;
|
||||
2. :func:`scripts.git_sync.clone_or_pull` each repo into
|
||||
``BOR_SOURCES_DIR/<repo-name>/`` (phase 28 — reused, not
|
||||
re-implemented; a failing repo aborts before any import);
|
||||
3. ``import_sources(..., prune=True)`` — prune so files deleted
|
||||
upstream leave the index (the CLI's no-prune default is unchanged);
|
||||
1. resolve the effective sources — the ``git_sources`` DB rows (git
|
||||
**and** local, phase 38), else the ``BOR_GIT_SOURCES`` fallback
|
||||
(git-only)
|
||||
(:func:`app.rag.git_sources.effective_sources`, shared with the
|
||||
CLI) — empty on both origins (no git rows, no local rows, no env
|
||||
URLs) fails loudly (``no sources configured (git or local)``)
|
||||
instead of silently importing the legacy local directories;
|
||||
2. per resolved row: ``kind=git`` → :func:`scripts.git_sync.clone_or_pull`
|
||||
into ``BOR_SOURCES_DIR/<repo-name>/`` (phase 28 — reused, not
|
||||
re-implemented); ``kind=local`` → the stored directory, re-verified
|
||||
``.is_dir()`` **at sync time** (it may have moved/deleted since
|
||||
add-time) — a missing directory raises ``local source missing:
|
||||
<path>``; a failing clone or a missing local dir aborts before any
|
||||
import;
|
||||
3. ``import_sources(..., prune=True)`` over the single combined list
|
||||
(git checkouts + local dirs) — prune so files deleted upstream or
|
||||
out of a local dir leave the index (pruning covers the union; the
|
||||
CLI's no-prune default is unchanged);
|
||||
4. when the import changed the KB (added + updated > 0),
|
||||
``regenerate_overview`` refreshes the single ``kb_overview`` row
|
||||
(phase 31 trigger, best-effort inside).
|
||||
@@ -48,7 +55,7 @@ from fastapi import APIRouter, Depends, HTTPException
|
||||
from app.config import get_settings
|
||||
from app.core.auth import require_admin
|
||||
from app.db import SessionLocal
|
||||
from app.rag.git_sources import effective_git_sources
|
||||
from app.rag.git_sources import effective_sources
|
||||
from app.rag.importer import ImportSummary, import_sources
|
||||
from app.rag.llm import LLMClient
|
||||
from app.rag.overview import regenerate_overview
|
||||
@@ -147,24 +154,41 @@ async def _run_sync() -> None:
|
||||
try:
|
||||
settings = get_settings()
|
||||
# The background task has no request session: open a short-lived
|
||||
# one around the shared phase-35 resolver (DB rows win, the
|
||||
# BOR_GIT_SOURCES list is a fallback while the table is empty).
|
||||
# one around the shared phase-35/38 resolver (DB rows of both
|
||||
# kinds win; the BOR_GIT_SOURCES git list is a fallback while
|
||||
# the table is empty).
|
||||
db = SessionLocal()
|
||||
try:
|
||||
git_urls, origin = effective_git_sources(db)
|
||||
rows, origin = effective_sources(db)
|
||||
finally:
|
||||
db.close()
|
||||
if not git_urls:
|
||||
# The button targets git sources only (manual --source dirs
|
||||
# have no repo to clone) — an empty config on *both* origins
|
||||
# fails loudly instead of silently importing the legacy
|
||||
# directories.
|
||||
raise GitSyncError(
|
||||
"no git sources configured (git_sources table empty and BOR_GIT_SOURCES unset)"
|
||||
if not rows:
|
||||
# The button targets the admin-managed source registry
|
||||
# (manual --source dirs have no repo to clone) — an empty
|
||||
# config on *both* origins (no git rows, no local rows, no
|
||||
# env URLs) fails loudly instead of silently importing the
|
||||
# legacy directories.
|
||||
raise GitSyncError("no sources configured (git or local)")
|
||||
git_count = sum(1 for row in rows if row.kind == "git")
|
||||
logger.info(
|
||||
"sync: started repos=%d origin=%s git=%d local=%d",
|
||||
len(rows), origin, git_count, len(rows) - git_count,
|
||||
)
|
||||
logger.info("sync: started repos=%d origin=%s", len(git_urls), origin)
|
||||
sources_root = Path(settings.sources_dir).expanduser()
|
||||
sources = [clone_or_pull(url, sources_root / repo_name(url)) for url in git_urls]
|
||||
sources: list[Path] = []
|
||||
for row in rows:
|
||||
if row.kind == "git":
|
||||
sources.append(clone_or_pull(row.url, sources_root / repo_name(row.url)))
|
||||
else:
|
||||
# kind=local — the stored expanded path (phase 38 also
|
||||
# mirrors it in the NOT-NULL ``url`` location column, the
|
||||
# ``or`` keeps the type checker honest); re-verified at
|
||||
# sync time because the directory may have moved or been
|
||||
# deleted since add-time.
|
||||
path = Path(row.path or row.url).expanduser()
|
||||
if not path.is_dir():
|
||||
raise GitSyncError(f"local source missing: {path}")
|
||||
sources.append(path)
|
||||
llm = LLMClient()
|
||||
summary: ImportSummary = await import_sources(sources, llm, prune=True)
|
||||
overview = False
|
||||
|
||||
+18
-7
@@ -13,8 +13,10 @@ Data model — see ``.agent/PLAN.md`` §Data Model:
|
||||
* ``kb_overview`` — single-row lite-generated outline of the KB's basic
|
||||
categories, injected as the ``<knowledge_base>``
|
||||
section of every chat turn (phase 31).
|
||||
* ``git_sources`` — admin-managed git source URLs the Sync button and
|
||||
import_docs clone/pull (phase 35).
|
||||
* ``git_sources`` — admin-managed source registry (git URLs + local
|
||||
directories) the Sync button and import_docs
|
||||
import (phase 35; ``kind`` discriminator added in
|
||||
phase 38).
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
@@ -138,16 +140,25 @@ class KbOverview(Base):
|
||||
|
||||
|
||||
class GitSource(Base):
|
||||
"""One admin-managed git source (phase 35).
|
||||
"""One admin-managed source (phase 35; kind discriminator, phase 38).
|
||||
|
||||
The UI-maintained list of repo URLs the Sync button (phase 32) and
|
||||
import_docs (phase 28) clone/pull. DB rows win over the
|
||||
BOR_GIT_SOURCES env var, which is a fallback while this table is
|
||||
empty (see app.rag.git_sources.effective_git_sources).
|
||||
The UI-maintained list the Sync button (phase 32) and import_docs
|
||||
(phase 28) import from. ``kind`` discriminates: ``git`` rows carry a
|
||||
repo ``url`` (cloned/pulled), ``local`` rows carry an existing
|
||||
directory ``path`` (walked directly). DB rows win over the
|
||||
BOR_GIT_SOURCES env var (git-only fallback), which is a fallback
|
||||
while this table is empty (see
|
||||
app.rag.git_sources.effective_git_sources).
|
||||
"""
|
||||
|
||||
__tablename__ = "git_sources"
|
||||
|
||||
id: Mapped[uuid.UUID] = mapped_column(UUID(as_uuid=True), primary_key=True, default=uuid.uuid4)
|
||||
url: Mapped[str] = mapped_column(Text, unique=True, nullable=False)
|
||||
#: Source-kind discriminator (phase 38): "git" (default) or "local"
|
||||
#: — enforced by the ``ck_git_sources_kind`` CHECK constraint.
|
||||
kind: Mapped[str] = mapped_column(Text, default="git", server_default="'git'")
|
||||
#: Absolute directory of a ``local`` source; NULL for git rows.
|
||||
#: Unique — Postgres treats NULLs as distinct under a unique index.
|
||||
path: Mapped[str | None] = mapped_column(Text, unique=True)
|
||||
added_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), server_default=func.now())
|
||||
|
||||
+45
-20
@@ -1,19 +1,26 @@
|
||||
"""The shared git-source resolver (phase 35, task 03).
|
||||
"""The shared source resolver (phase 35, task 03; local kind, phase 38).
|
||||
|
||||
One function, two callers: the in-app Sync pipeline
|
||||
(``app/api/sync.py::_run_sync``) and the CLI
|
||||
(``scripts/import_docs.py::_resolve_sources``) — both resolve the repo
|
||||
URLs to clone/pull through :func:`effective_git_sources`, so the
|
||||
admin-managed ``git_sources`` table is what actually gets cloned and
|
||||
indexed from either entry point (the API's ``GET /api/git-sources``
|
||||
fallback list is the only other place that reads the env list — task 02).
|
||||
(``scripts/import_docs.py::_resolve_sources``) — both resolve the
|
||||
``git_sources`` rows to import through :func:`effective_sources`, so
|
||||
the admin-managed ``git_sources`` table is what actually gets cloned
|
||||
(git rows) or walked directly (local rows) from either entry point
|
||||
(the API's ``GET /api/git-sources`` fallback list is the only other
|
||||
place that reads the env list — task 02).
|
||||
|
||||
Precedence (the phase's locked decision — the env var is demoted, not
|
||||
removed): ``git_sources`` DB rows win, in ``(added_at, id)`` order;
|
||||
``BOR_GIT_SOURCES`` is a fallback only while the table is empty; both
|
||||
empty → ``([], "env")`` and each caller keeps its existing fail-loud
|
||||
behavior (sync: ``GitSyncError``; CLI: the legacy ``DEFAULT_SOURCES``
|
||||
fallback).
|
||||
removed): ``git_sources`` DB rows of **both kinds** win, in
|
||||
``(added_at, id)`` order; ``BOR_GIT_SOURCES`` is a fallback only while
|
||||
the table is empty (git URLs surface as synthetic ``kind='git'`` rows —
|
||||
the env fallback is git-only); both empty → ``([], "env")`` and each
|
||||
caller keeps its existing fail-loud behavior (sync: ``GitSyncError``;
|
||||
CLI: the legacy ``DEFAULT_SOURCES`` fallback).
|
||||
|
||||
:func:`effective_git_sources` is kept as the phase-35 back-compat alias
|
||||
(repo URLs of the effective git rows) so existing importers of the old
|
||||
name keep working; new code calls :func:`effective_sources` and
|
||||
branches on ``row.kind``.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
@@ -26,19 +33,37 @@ from app.config import get_settings
|
||||
from app.models import GitSource
|
||||
|
||||
|
||||
def effective_git_sources(db: Session) -> tuple[list[str], Literal["db", "env"]]:
|
||||
"""``(urls, origin)`` — the repo URLs to clone, and where they came from.
|
||||
def effective_sources(db: Session) -> tuple[list[GitSource], Literal["db", "env"]]:
|
||||
"""``(rows, origin)`` — the effective source rows (both kinds) and
|
||||
where they came from.
|
||||
|
||||
``"db"``: the ``git_sources`` rows in ``(added_at, id)`` order (oldest
|
||||
first, id tie-break for same-timestamp inserts) — once the table has
|
||||
any row, ``BOR_GIT_SOURCES`` is ignored entirely.
|
||||
``"db"``: the ``git_sources`` rows — ``kind='git'`` (repo URL to
|
||||
clone/pull) and ``kind='local'`` (existing directory to walk
|
||||
directly) — in ``(added_at, id)`` order (oldest first, id tie-break
|
||||
for same-timestamp inserts). Once the table has any row,
|
||||
``BOR_GIT_SOURCES`` is ignored entirely.
|
||||
``"env"``: the ``BOR_GIT_SOURCES`` list — ``Settings.git_source_list``,
|
||||
the phase-28 CSV parse, reused not re-implemented — used only while
|
||||
the table is empty; both empty → ``([], "env")``.
|
||||
the phase-28 CSV parse, reused not re-implemented — surfaced as
|
||||
synthetic ``kind='git'`` rows (``path`` NULL; the env fallback is
|
||||
git-only), used only while the table is empty; both empty →
|
||||
``([], "env")``.
|
||||
"""
|
||||
rows = db.scalars(
|
||||
select(GitSource).order_by(GitSource.added_at.asc(), GitSource.id.asc())
|
||||
).all()
|
||||
if rows:
|
||||
return [row.url for row in rows], "db"
|
||||
return list(get_settings().git_source_list), "env"
|
||||
return list(rows), "db"
|
||||
return [GitSource(url=url, kind="git") for url in get_settings().git_source_list], "env"
|
||||
|
||||
|
||||
def effective_git_sources(db: Session) -> tuple[list[str], Literal["db", "env"]]:
|
||||
"""Back-compat alias for the phase-35 name (existing importers).
|
||||
|
||||
Returns the repo URLs of the effective ``kind='git'`` rows only, with
|
||||
the same origin label — the env fallback is git-only, so its
|
||||
behavior is unchanged. Local rows carry no URL (they are walked
|
||||
directly); new code should call :func:`effective_sources` and branch
|
||||
on ``row.kind``.
|
||||
"""
|
||||
rows, origin = effective_sources(db)
|
||||
return [row.url for row in rows if row.kind == "git"], origin
|
||||
|
||||
+51
-12
@@ -181,28 +181,47 @@ class SteeringNoteList(BaseModel):
|
||||
|
||||
|
||||
class GitSourceIn(BaseModel):
|
||||
"""``POST /api/git-sources`` body: one repo URL (phase 35, task 02).
|
||||
"""``POST /api/git-sources`` body (phase 35, task 02; ``kind``, phase 38).
|
||||
|
||||
Mirrors :class:`SteeringNoteIn` — the URL is trimmed *before* the
|
||||
length constraints run, so a whitespace-only body is a 422 and a URL
|
||||
with surrounding spaces is stored clean. Shape validation
|
||||
(``https://``, ``ssh://``, ``git@``) happens in the API layer so the
|
||||
422 detail can be one generic string that never echoes the input.
|
||||
``kind`` selects the source kind and which field carries its location:
|
||||
|
||||
* ``"git"`` (default) — ``url`` is the repo URL. Mirrors the
|
||||
phase-35 contract: trimmed *before* the length constraints run, so a
|
||||
whitespace-only body is a 422 and a URL with surrounding spaces is
|
||||
stored clean. Shape validation (``https://``, ``ssh://``, ``git@``)
|
||||
and the kind-field rules (url present, no path) happen in the API
|
||||
layer so the 422/409 details stay fixed strings that never echo the
|
||||
input (credential safety).
|
||||
* ``"local"`` — ``path`` is an existing directory on the server.
|
||||
Trimmed here; the API layer then ``expanduser()``s it and requires an
|
||||
absolute existing directory (else 422 naming the path — the path is
|
||||
not a secret, unlike a git URL) and no ``url``.
|
||||
"""
|
||||
|
||||
url: str = Field(min_length=1, max_length=500)
|
||||
kind: Literal["git", "local"] = "git"
|
||||
url: str | None = Field(default=None, min_length=1, max_length=500)
|
||||
path: str | None = Field(default=None, min_length=1, max_length=2000)
|
||||
|
||||
@field_validator("url", mode="before")
|
||||
@classmethod
|
||||
def _trim_url(cls, v: object) -> object:
|
||||
return v.strip() if isinstance(v, str) else v
|
||||
|
||||
@field_validator("path", mode="before")
|
||||
@classmethod
|
||||
def _trim_path(cls, v: object) -> object:
|
||||
return v.strip() if isinstance(v, str) else v
|
||||
|
||||
|
||||
class GitSourceOut(BaseModel):
|
||||
"""One git source as returned by the API (phase 35, task 02).
|
||||
"""One created git source as returned by ``POST`` (phase 35, task 02).
|
||||
|
||||
``id`` / ``added_at`` are nullable: env-fallback rows (table empty →
|
||||
the list comes from ``BOR_GIT_SOURCES``) carry neither, only a URL.
|
||||
``id`` / ``added_at`` are non-null for a stored row. ``url`` is the
|
||||
row's location column: the repo URL for ``kind=git`` rows and, for
|
||||
``kind=local`` rows, the stored (expanded) directory path — the
|
||||
phase-35 response shape is unchanged by phase 38, so a local 201
|
||||
reports its path in ``url`` and the full row (``kind`` + ``path``)
|
||||
via ``GET``.
|
||||
"""
|
||||
|
||||
id: uuid.UUID | None
|
||||
@@ -210,14 +229,34 @@ class GitSourceOut(BaseModel):
|
||||
added_at: datetime | None
|
||||
|
||||
|
||||
class GitSourceRow(BaseModel):
|
||||
"""One row of ``GET /api/git-sources`` (phase 35; ``kind``/``path``,
|
||||
phase 38, task 02).
|
||||
|
||||
``kind`` discriminates the row: git rows (and the git-only
|
||||
``BOR_GIT_SOURCES`` env-fallback rows) carry ``url`` and
|
||||
``path: null``; local rows carry ``path`` (the absolute directory,
|
||||
expanded) and the same string in ``url`` (the table's NOT-NULL
|
||||
location column). ``id`` / ``added_at`` are nullable: env-fallback
|
||||
rows (table empty) carry neither.
|
||||
"""
|
||||
|
||||
id: uuid.UUID | None
|
||||
kind: Literal["git", "local"]
|
||||
url: str
|
||||
path: str | None
|
||||
added_at: datetime | None
|
||||
|
||||
|
||||
class GitSourceList(BaseModel):
|
||||
"""``GET /api/git-sources`` response (phase 35, task 02).
|
||||
|
||||
``from_env`` is True only when the ``git_sources`` table is empty and
|
||||
the list comes from ``BOR_GIT_SOURCES`` (the phase's locked fallback);
|
||||
the list comes from ``BOR_GIT_SOURCES`` (the phase's locked fallback
|
||||
— env rows are git-only and report ``kind: "git"``, ``path: null``);
|
||||
once the table has rows the env var is ignored and ``from_env`` is
|
||||
False — the UI is the source of truth.
|
||||
"""
|
||||
|
||||
sources: list[GitSourceOut]
|
||||
sources: list[GitSourceRow]
|
||||
from_env: bool
|
||||
|
||||
+118
-54
@@ -1,7 +1,10 @@
|
||||
/* Brain of Reese — Git sources admin page (phase 35, task 04).
|
||||
/* Brain of Reese — Git sources admin page (phase 35, task 04;
|
||||
* local directories, phase 38 task 04).
|
||||
*
|
||||
* The page module for /git-sources.html: the admin-only manager for the
|
||||
* stored git source list (git-sources table, phase 35 tasks 01/02).
|
||||
* stored source list (git-sources table, phase 35 tasks 01/02) — git
|
||||
* repo URLs (kind "git") and existing local directories (kind
|
||||
* "local", phase 38).
|
||||
* This module is the single owner of the page's behaviour:
|
||||
*
|
||||
* • boot — initSharedHeader() (one cached whoami, shared with the
|
||||
@@ -11,19 +14,25 @@
|
||||
* #git-sources-content revealed, loadSources().
|
||||
* • loadSources() — GET /api/git-sources → the table rows
|
||||
* (#git-sources-tbody), the env-fallback note's visibility
|
||||
* (from_env), and the empty state. URLs are ALWAYS rendered with
|
||||
* textContent — never innerHTML (they may embed user:pass@
|
||||
* (from_env), and the empty state. Each row leads with its kind
|
||||
* badge (Git/Local — text + color, never color alone) plus the
|
||||
* location in a mono <code>: the git URL, or the full local path
|
||||
* for kind "local" rows (phase 38). Values are ALWAYS rendered
|
||||
* with textContent — never innerHTML (URLs may embed user:pass@
|
||||
* credentials; phase 32's masking discipline). Non-2xx or a
|
||||
* network failure renders the role="alert" load error with a
|
||||
* retry button — never a stuck page.
|
||||
* • add — #git-source-form submit → POST /api/git-sources {url}.
|
||||
* §7.4 never-stale: the button disables + relabels "Adding…"
|
||||
* while the request is out, re-enables ("Add source") on success
|
||||
* AND failure. 201 clears the input, reloads the list, and
|
||||
* focuses the new row's Remove button (a11y); a failure (409
|
||||
* duplicate, 422 shape) shows the server detail inline under the
|
||||
* form (role="alert", 422 shape-aware like the tuning forms) and
|
||||
* keeps the input — the instruction survives.
|
||||
* • add — #git-source-form submit → POST /api/git-sources {url};
|
||||
* #local-source-form submit → POST /api/git-sources
|
||||
* {kind: "local", path} (phase 38). ONE §7.4 never-stale
|
||||
* lifecycle for both (wireAddForm): the button disables +
|
||||
* relabels "Adding…" while the request is out, re-enables on
|
||||
* success AND failure. 201 clears the input, reloads the list,
|
||||
* and focuses the new row's Remove button (a11y); a failure (409
|
||||
* duplicate, 422 validation) shows the server detail inline under
|
||||
* the form (role="alert", 422 shape-aware like the tuning forms)
|
||||
* and keeps the input — the instruction survives. Local 422/409
|
||||
* details name the path (paths are not secrets, unlike URLs).
|
||||
* • remove — a row's Remove button asks window.confirm first
|
||||
* (removal prunes the documents only on the NEXT sync — the
|
||||
* confirm says so). Cancel → nothing; ok → the row button
|
||||
@@ -56,6 +65,12 @@ const formEl = document.querySelector("#git-source-form");
|
||||
const urlInput = document.querySelector("#git-source-url");
|
||||
const addBtn = document.querySelector("#git-source-add");
|
||||
const addError = document.querySelector("#git-source-error");
|
||||
/* Phase 38: the second add form — "Local directory" (same element
|
||||
contract as the git form, own ids). */
|
||||
const localFormEl = document.querySelector("#local-source-form");
|
||||
const pathInput = document.querySelector("#local-source-path");
|
||||
const localAddBtn = document.querySelector("#local-source-add");
|
||||
const localAddError = document.querySelector("#local-source-error");
|
||||
const loadErrorEl = document.querySelector("#git-sources-load-error");
|
||||
const loadErrorText = document.querySelector("#git-sources-load-error-text");
|
||||
const retryBtn = document.querySelector("#git-sources-retry");
|
||||
@@ -125,7 +140,7 @@ async function loadSources() {
|
||||
hideLoadError();
|
||||
const sources = Array.isArray(data.sources) ? data.sources : [];
|
||||
renderSources(sources, data.from_env === true);
|
||||
announce(`${sources.length} git source${sources.length === 1 ? "" : "s"} listed.`);
|
||||
announce(`${sources.length} source${sources.length === 1 ? "" : "s"} listed.`);
|
||||
}
|
||||
|
||||
function showLoadError(message) {
|
||||
@@ -153,11 +168,15 @@ function renderSources(sources, fromEnv) {
|
||||
if (emptyEl) emptyEl.hidden = hasRows;
|
||||
}
|
||||
|
||||
/* One row: the URL in a mono <code> (textContent only — URLs may
|
||||
contain credentials), the localized added date ("—" for env
|
||||
fallback rows), and the per-row Remove button — or the "from .env"
|
||||
tag for env-fallback rows (id null: nothing is stored to remove;
|
||||
the env note says where the active list comes from). */
|
||||
/* One row: the kind badge (Git/Local — phase 38) followed by the
|
||||
location in a mono <code> (textContent only — git URLs may contain
|
||||
credentials, local paths may contain anything), the localized added
|
||||
date ("—" for env fallback rows), and the per-row Remove button —
|
||||
or the "from .env" tag for env-fallback rows (id null: nothing is
|
||||
stored to remove; the env note says where the active list comes
|
||||
from). The value is the git URL for kind "git" rows and the full
|
||||
local path for kind "local" rows (the API reports the path in both
|
||||
`path` and `url`; `path` is the kind-typed field). */
|
||||
const REMOVE_ICON =
|
||||
'<svg aria-hidden="true" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="1.8" stroke-linecap="round"><path d="M5 7h14M10 7V5h4v2M8.5 7l.7 12h5.6l.7-12"/></svg>';
|
||||
|
||||
@@ -165,12 +184,19 @@ function makeRow(s) {
|
||||
const tr = document.createElement("tr");
|
||||
if (s.id) tr.dataset.id = s.id;
|
||||
|
||||
const isLocal = s.kind === "local";
|
||||
const value = isLocal ? (s.path ?? s.url) : s.url;
|
||||
const kindLabel = isLocal ? "local" : "git";
|
||||
|
||||
const urlTd = document.createElement("td");
|
||||
urlTd.className = "git-source-url-cell";
|
||||
urlTd.title = s.url; // full URL on hover (long URLs scroll the wrapper)
|
||||
urlTd.title = value; // full URL/path on hover (long values scroll the wrapper)
|
||||
const badge = document.createElement("span");
|
||||
badge.className = `git-source-kind is-${isLocal ? "local" : "git"}`;
|
||||
badge.textContent = isLocal ? "Local" : "Git"; // text + color, never color alone
|
||||
const code = document.createElement("code");
|
||||
code.textContent = s.url; // rendered as text, never as HTML
|
||||
urlTd.appendChild(code);
|
||||
code.textContent = value; // rendered as text, never as HTML
|
||||
urlTd.append(badge, code);
|
||||
tr.appendChild(urlTd);
|
||||
|
||||
const addedTd = document.createElement("td");
|
||||
@@ -183,13 +209,13 @@ function makeRow(s) {
|
||||
const btn = document.createElement("button");
|
||||
btn.type = "button";
|
||||
btn.className = "git-source-remove";
|
||||
btn.setAttribute("aria-label", `Remove git source: ${s.url}`);
|
||||
btn.setAttribute("aria-label", `Remove ${kindLabel} source: ${value}`);
|
||||
btn.innerHTML = REMOVE_ICON + "<span>Remove</span>";
|
||||
const rowError = document.createElement("span");
|
||||
rowError.className = "git-source-row-error";
|
||||
rowError.setAttribute("role", "alert");
|
||||
rowError.hidden = true;
|
||||
btn.addEventListener("click", () => removeSource(s, btn, rowError));
|
||||
btn.addEventListener("click", () => removeSource(s, btn, rowError, kindLabel));
|
||||
actTd.append(btn, rowError);
|
||||
} else {
|
||||
const tag = document.createElement("span");
|
||||
@@ -206,9 +232,9 @@ function makeRow(s) {
|
||||
* (phase scope boundary), so the confirm says exactly that. Cancel →
|
||||
* nothing; a failed delete → per-row role="alert" error + re-enabled
|
||||
* button (never a stuck row); success → the list reloads. */
|
||||
async function removeSource(s, btn, rowError) {
|
||||
async function removeSource(s, btn, rowError, kindLabel) {
|
||||
const ok = window.confirm(
|
||||
"Remove this git source from the list? Its documents stay indexed until the next sync prunes them.",
|
||||
`Remove this ${kindLabel} source from the list? Its documents stay indexed until the next sync prunes them.`,
|
||||
);
|
||||
if (!ok) return;
|
||||
btn.disabled = true; // one delete per click
|
||||
@@ -216,43 +242,55 @@ async function removeSource(s, btn, rowError) {
|
||||
try {
|
||||
const r = await fetch(`/api/git-sources/${encodeURIComponent(s.id)}`, { method: "DELETE" });
|
||||
if (!r.ok) {
|
||||
rowError.textContent = await apiDetail(r, "Could not remove the git source — try again.");
|
||||
rowError.textContent = await apiDetail(r, `Could not remove the ${kindLabel} source — try again.`);
|
||||
rowError.hidden = false;
|
||||
btn.disabled = false;
|
||||
return;
|
||||
}
|
||||
announce("Git source removed.");
|
||||
announce("Source removed.");
|
||||
await loadSources(); // 204: the server confirmed — the list re-renders
|
||||
} catch {
|
||||
rowError.textContent = "Could not remove the git source — is the app reachable?";
|
||||
rowError.textContent = `Could not remove the ${kindLabel} source — is the app reachable?`;
|
||||
rowError.hidden = false;
|
||||
btn.disabled = false;
|
||||
}
|
||||
}
|
||||
|
||||
/* ---------- add (POST /api/git-sources) ---------- */
|
||||
|
||||
if (formEl && urlInput && addBtn) {
|
||||
formEl.addEventListener("submit", async (e) => {
|
||||
/* ---------- add (POST /api/git-sources) — both forms, one lifecycle
|
||||
* (the local form is phase 38) ----------
|
||||
* The git form posts {url}; the local form posts {kind:"local",path}.
|
||||
* wireAddForm gives both the §7.4 never-stale lifecycle: while the
|
||||
* request is out the button disables + relabels "Adding…" and
|
||||
* re-enables (idle label restored) on success AND failure. 201 clears
|
||||
* the input, reloads the list, and focuses the new row's Remove button
|
||||
* (a11y); a failure (409 duplicate, 422 validation) shows the server
|
||||
* detail inline under the form (role="alert", 422 shape-aware via
|
||||
* apiDetail) and keeps the input — the fix is one edit, not a re-type.
|
||||
* Git 409/422 details are fixed generic strings (credential safety);
|
||||
* local details name the path (not a secret). */
|
||||
function wireAddForm(opts) {
|
||||
const { form, input, btn, error } = opts;
|
||||
if (!form || !input || !btn) return;
|
||||
form.addEventListener("submit", async (e) => {
|
||||
e.preventDefault();
|
||||
// Client-side non-empty check (the input is `required` too — the
|
||||
// browser's native prompt is the first line, this one the second).
|
||||
const url = urlInput.value.trim();
|
||||
if (!url) {
|
||||
if (addError) {
|
||||
addError.textContent = "Enter a git URL to add.";
|
||||
addError.hidden = false;
|
||||
const value = input.value.trim();
|
||||
if (!value) {
|
||||
if (error) {
|
||||
error.textContent = opts.emptyMessage;
|
||||
error.hidden = false;
|
||||
}
|
||||
return;
|
||||
}
|
||||
if (addError) addError.hidden = true; // a new attempt starts clean
|
||||
addBtn.disabled = true; // §7.4: one POST per click
|
||||
addBtn.textContent = "Adding…";
|
||||
if (error) error.hidden = true; // a new attempt starts clean
|
||||
btn.disabled = true; // §7.4: one POST per click
|
||||
btn.textContent = "Adding…";
|
||||
try {
|
||||
const r = await fetch("/api/git-sources", {
|
||||
method: "POST",
|
||||
headers: { "Content-Type": "application/json" },
|
||||
body: JSON.stringify({ url }),
|
||||
body: JSON.stringify(opts.body(value)),
|
||||
});
|
||||
if (r.ok) {
|
||||
let createdId = null;
|
||||
@@ -261,31 +299,57 @@ if (formEl && urlInput && addBtn) {
|
||||
} catch {
|
||||
/* the 201 body is advisory — the reload is the truth */
|
||||
}
|
||||
urlInput.value = ""; // 201: the source is stored
|
||||
announce("Git source added.");
|
||||
input.value = ""; // 201: the source is stored
|
||||
announce(opts.addedMessage);
|
||||
await loadSources(); // the new row lands in the table
|
||||
focusNewRow(createdId); // a11y: land the caret on the new row
|
||||
return;
|
||||
}
|
||||
// 409 duplicate / 422 shape / anything else: the server detail
|
||||
// inline (never echoing a URL the server wouldn't), form kept —
|
||||
// the input survives so the fix is one edit, not a re-type.
|
||||
if (addError) {
|
||||
addError.textContent = await apiDetail(r, "Could not add the git source — try again.");
|
||||
addError.hidden = false;
|
||||
// 409 duplicate / 422 validation / anything else: the server
|
||||
// detail inline, form kept — the input survives so the fix is
|
||||
// one edit, not a re-type.
|
||||
if (error) {
|
||||
error.textContent = await apiDetail(r, opts.failMessage);
|
||||
error.hidden = false;
|
||||
}
|
||||
} catch {
|
||||
if (addError) {
|
||||
addError.textContent = "Could not add the git source — is the app reachable?";
|
||||
addError.hidden = false;
|
||||
if (error) {
|
||||
error.textContent = opts.networkMessage;
|
||||
error.hidden = false;
|
||||
}
|
||||
} finally {
|
||||
addBtn.disabled = false; // never stale — success OR failure
|
||||
addBtn.textContent = "Add source";
|
||||
btn.disabled = false; // never stale — success OR failure
|
||||
btn.textContent = opts.idleLabel;
|
||||
}
|
||||
});
|
||||
}
|
||||
|
||||
wireAddForm({
|
||||
form: formEl,
|
||||
input: urlInput,
|
||||
btn: addBtn,
|
||||
error: addError,
|
||||
body: (url) => ({ url }),
|
||||
emptyMessage: "Enter a git URL to add.",
|
||||
failMessage: "Could not add the git source — try again.",
|
||||
networkMessage: "Could not add the git source — is the app reachable?",
|
||||
addedMessage: "Git source added.",
|
||||
idleLabel: "Add source",
|
||||
});
|
||||
|
||||
wireAddForm({
|
||||
form: localFormEl,
|
||||
input: pathInput,
|
||||
btn: localAddBtn,
|
||||
error: localAddError,
|
||||
body: (path) => ({ kind: "local", path }),
|
||||
emptyMessage: "Enter a directory path to add.",
|
||||
failMessage: "Could not add the local directory — try again.",
|
||||
networkMessage: "Could not add the local directory — is the app reachable?",
|
||||
addedMessage: "Local source added.",
|
||||
idleLabel: "Add directory",
|
||||
});
|
||||
|
||||
/* After a successful add, focus the new row's Remove button so the
|
||||
keyboard/screen-reader user lands where the new data is. The 201
|
||||
body carries the row id (tr[data-id]); without one, the first row
|
||||
|
||||
+58
-22
@@ -1331,11 +1331,15 @@ details.thinking .thinking-text ul { margin: 0 0 0.5rem; }
|
||||
flex: 1;
|
||||
}
|
||||
|
||||
/* Add form — the tuning form's surface as a single row: visible label
|
||||
+ mono URL input (the credentials case is real, so the input is
|
||||
mono) + the brand "Add source" button; wraps to a column at narrow
|
||||
widths (the <=640px block below). */
|
||||
#git-source-form {
|
||||
/* Add forms — the tuning form's surface as a single row: visible
|
||||
label + mono location input (git URLs may embed credentials, local
|
||||
paths may contain anything, so both inputs are mono) + the brand
|
||||
button; wraps to a column at narrow widths (the <=640px block
|
||||
below). Phase 38: the "Local directory" form (#local-source-form)
|
||||
reuses the git form's rules VERBATIM — one form language for both
|
||||
kinds. */
|
||||
#git-source-form,
|
||||
#local-source-form {
|
||||
display: flex;
|
||||
flex-wrap: wrap;
|
||||
align-items: center;
|
||||
@@ -1346,9 +1350,12 @@ details.thinking .thinking-text ul { margin: 0 0 0.5rem; }
|
||||
box-shadow: var(--shadow);
|
||||
padding: 0.9rem 1rem 1rem;
|
||||
}
|
||||
#git-source-form:focus-within { border-color: var(--brand); box-shadow: 0 0 0 3px var(--brand-soft), var(--shadow); }
|
||||
#git-source-form > label { color: var(--ink); font-weight: 600; white-space: nowrap; }
|
||||
#git-source-url {
|
||||
#git-source-form:focus-within,
|
||||
#local-source-form:focus-within { border-color: var(--brand); box-shadow: 0 0 0 3px var(--brand-soft), var(--shadow); }
|
||||
#git-source-form > label,
|
||||
#local-source-form > label { color: var(--ink); font-weight: 600; white-space: nowrap; }
|
||||
#git-source-url,
|
||||
#local-source-path {
|
||||
flex: 1;
|
||||
min-width: 14rem;
|
||||
min-height: 44px;
|
||||
@@ -1360,9 +1367,12 @@ details.thinking .thinking-text ul { margin: 0 0 0.5rem; }
|
||||
border-radius: var(--radius-sm);
|
||||
padding: 0.45rem 0.7rem;
|
||||
}
|
||||
#git-source-url::placeholder { color: var(--ink-soft); }
|
||||
#git-source-url:focus-visible { outline-offset: 0; border-color: var(--brand); }
|
||||
#git-source-add {
|
||||
#git-source-url::placeholder,
|
||||
#local-source-path::placeholder { color: var(--ink-soft); }
|
||||
#git-source-url:focus-visible,
|
||||
#local-source-path:focus-visible { outline-offset: 0; border-color: var(--brand); }
|
||||
#git-source-add,
|
||||
#local-source-add {
|
||||
display: inline-flex;
|
||||
align-items: center;
|
||||
justify-content: center;
|
||||
@@ -1376,8 +1386,10 @@ details.thinking .thinking-text ul { margin: 0 0 0.5rem; }
|
||||
font-weight: 700;
|
||||
cursor: pointer;
|
||||
}
|
||||
#git-source-add:hover:not(:disabled) { background: #7d88f5; }
|
||||
#git-source-add:disabled { opacity: 0.6; cursor: wait; }
|
||||
#git-source-add:hover:not(:disabled),
|
||||
#local-source-add:hover:not(:disabled) { background: #7d88f5; }
|
||||
#git-source-add:disabled,
|
||||
#local-source-add:disabled { opacity: 0.6; cursor: wait; }
|
||||
|
||||
/* The add form's inline error (role=alert): the err pair (9.1:1);
|
||||
flex-basis 100% drops it onto its own row under the input. */
|
||||
@@ -1491,10 +1503,29 @@ details.thinking .thinking-text ul { margin: 0 0 0.5rem; }
|
||||
position: sticky;
|
||||
top: 0;
|
||||
}
|
||||
/* URL cell: mono at the Sources-table size; the <code> is plain
|
||||
(no chip background — the cell IS the mono readout). */
|
||||
/* Location cell: mono at the Sources-table size; the <code> is plain
|
||||
(no chip background — the cell IS the mono readout). Phase 38:
|
||||
leads with the kind badge (Git/Local) — the kinds are told apart by
|
||||
TEXT as much as color (WCAG 1.4.1), and both badge pairs are AA:
|
||||
brand-ink on brand-soft 6.9:1 (Git), ok-ink on ok-bg 10.6:1
|
||||
(Local). */
|
||||
.git-sources-table td.git-source-url-cell { font-family: var(--mono); font-size: 0.82rem; }
|
||||
.git-sources-table td.git-source-url-cell code { font-family: inherit; }
|
||||
.git-source-kind {
|
||||
display: inline-block;
|
||||
margin-right: 0.55rem;
|
||||
padding: 0.08rem 0.5rem;
|
||||
border-radius: 999px;
|
||||
font-family: var(--font);
|
||||
font-size: 0.72rem;
|
||||
font-weight: 700;
|
||||
letter-spacing: 0.04em;
|
||||
text-transform: uppercase;
|
||||
vertical-align: 0.06em;
|
||||
white-space: nowrap;
|
||||
}
|
||||
.git-source-kind.is-git { color: var(--brand-ink); background: var(--brand-soft); }
|
||||
.git-source-kind.is-local { color: var(--ok-ink); background: var(--ok-bg); }
|
||||
.git-sources-table tbody tr:hover { background: var(--bg); }
|
||||
.git-sources-table tbody tr:last-child td { border-bottom: 0; }
|
||||
|
||||
@@ -2102,13 +2133,18 @@ details.thinking .thinking-text ul { margin: 0 0 0.5rem; }
|
||||
.doc-modal-meta { padding-inline: 0.9rem; }
|
||||
.doc-modal-content { padding: 0.75rem 0.9rem 1.25rem; }
|
||||
.composer { padding: 0.5rem; }
|
||||
/* Phase 35: the git sources add form stacks like the other cards —
|
||||
label, full-width mono input, full-width button; the table
|
||||
wrapper's horizontal scroll already covers long URLs. */
|
||||
#git-source-form { flex-direction: column; align-items: stretch; }
|
||||
#git-source-form > label { white-space: normal; }
|
||||
#git-source-url { min-width: 0; }
|
||||
#git-source-add { width: 100%; }
|
||||
/* Phase 35 (phase 38: + the local directory form): the add forms
|
||||
stack like the other cards — label, full-width mono input,
|
||||
full-width button; the table wrapper's horizontal scroll already
|
||||
covers long URLs/paths. */
|
||||
#git-source-form,
|
||||
#local-source-form { flex-direction: column; align-items: stretch; }
|
||||
#git-source-form > label,
|
||||
#local-source-form > label { white-space: normal; }
|
||||
#git-source-url,
|
||||
#local-source-path { min-width: 0; }
|
||||
#git-source-add,
|
||||
#local-source-add { width: 100%; }
|
||||
.footer-inner { flex-direction: column; gap: 0.2rem; text-align: center; }
|
||||
main { padding-bottom: env(safe-area-inset-bottom, 0); }
|
||||
}
|
||||
|
||||
+39
-12
@@ -133,8 +133,9 @@
|
||||
<div class="page-head">
|
||||
<h1>Git sources</h1>
|
||||
<p class="page-sub">
|
||||
The repositories the Sync button clones and indexes. Add or
|
||||
remove them here — no <code>.env</code>, no restart.
|
||||
The git repositories and local directories the Sync button
|
||||
imports. Add or remove them here — no <code>.env</code>, no
|
||||
restart.
|
||||
</p>
|
||||
</div>
|
||||
|
||||
@@ -178,12 +179,36 @@
|
||||
<p class="git-source-error" id="git-source-error" role="alert" hidden></p>
|
||||
</form>
|
||||
|
||||
<div class="table-wrap" id="git-sources-table-wrap" role="region" aria-label="Git sources" tabindex="0">
|
||||
<!-- Phase 38: the second add form — "Local directory": an
|
||||
existing directory on the server (NOT a git repo), walked
|
||||
directly by Sync / import_docs. The SAME never-stale-button
|
||||
+ inline-error pattern as the git form (PLAN §7.4): the
|
||||
button disables + relabels "Adding…" while the POST is out
|
||||
and recovers on success AND failure; on success the input
|
||||
clears and the list re-fetches (the new row lands with the
|
||||
Local badge). A missing/relative path 422s with the path
|
||||
named inline (paths are not secrets, unlike git URLs). -->
|
||||
<form id="local-source-form">
|
||||
<label for="local-source-path">Add a local directory</label>
|
||||
<input
|
||||
id="local-source-path"
|
||||
name="path"
|
||||
type="text"
|
||||
maxlength="2000"
|
||||
autocomplete="off"
|
||||
placeholder="~/Notes"
|
||||
required
|
||||
>
|
||||
<button type="submit" id="local-source-add">Add directory</button>
|
||||
<p class="git-source-error" id="local-source-error" role="alert" hidden></p>
|
||||
</form>
|
||||
|
||||
<div class="table-wrap" id="git-sources-table-wrap" role="region" aria-label="Sources" tabindex="0">
|
||||
<table class="git-sources-table" id="git-sources-table">
|
||||
<caption class="visually-hidden">Git repositories the Sync button clones and indexes</caption>
|
||||
<caption class="visually-hidden">Sources the Sync button imports — git repositories it clones and local directories it walks</caption>
|
||||
<thead>
|
||||
<tr>
|
||||
<th scope="col">URL</th>
|
||||
<th scope="col">Source</th>
|
||||
<th scope="col">Added</th>
|
||||
<th scope="col">Actions</th>
|
||||
</tr>
|
||||
@@ -195,16 +220,18 @@
|
||||
<!-- Empty state — no stored rows AND no env fallback. With
|
||||
from_env, the env note above already explains where the
|
||||
active list comes from. -->
|
||||
<p class="git-sources-empty" id="git-sources-empty" hidden>No git sources stored yet.</p>
|
||||
<p class="git-sources-empty" id="git-sources-empty" hidden>No sources stored yet.</p>
|
||||
|
||||
<!-- Scope boundary (phase locked decision): adding/removing a
|
||||
repo does NOT clone or prune — the Sync button performs
|
||||
that. The hint says so. -->
|
||||
source does NOT clone or prune — the Sync button performs
|
||||
that. The hint says so (phase 38: git + local together,
|
||||
files removed from a source pruned). -->
|
||||
<p class="git-source-hint" id="git-sources-hint" role="note">
|
||||
Use the <strong>Sync sources</strong> button in the header (or on
|
||||
the Sources page) to clone the repos and refresh the index —
|
||||
removing a repository prunes its documents from the index on the
|
||||
next sync.
|
||||
Sync clones/pulls the git repos and imports the local
|
||||
directories together (files removed from a source are
|
||||
pruned). Use the <strong>Sync sources</strong> button in the
|
||||
header (or on the Sources page) to run it — removing a source
|
||||
prunes its documents from the index on the next sync.
|
||||
</p>
|
||||
</div>
|
||||
<!-- Polite live region: the screen-reader confirmation for list
|
||||
|
||||
+55
-28
@@ -7,20 +7,24 @@ Examples::
|
||||
uv run python -m scripts.import_docs --prune # also drop deleted/out-of-scope files
|
||||
uv run python -m scripts.import_docs --limit 5 # debug: first 5 files only
|
||||
|
||||
Source resolution (phase 28, extended in phase 35), in precedence order:
|
||||
Source resolution (phase 28, extended in phases 35 and 38), in
|
||||
precedence order:
|
||||
|
||||
1. ``--source PATH`` — explicit manual directories (repeatable) always win;
|
||||
git sources are ignored when this flag is used.
|
||||
2. The effective git sources — the admin-managed ``git_sources`` table
|
||||
rows, else the ``BOR_GIT_SOURCES`` (comma-separated) fallback
|
||||
(:func:`app.rag.git_sources.effective_git_sources`, the same shared
|
||||
resolver the in-app Sync button uses) — each repo is cloned (first
|
||||
run, shallow ``--depth 1``) or fast-forwarded (``git pull --ff-only``)
|
||||
into ``BOR_SOURCES_DIR/<repo-name>/`` (default ``~/bor-sources``) and
|
||||
the resulting checkouts are imported. A failing clone/pull aborts the
|
||||
whole run *before* anything is imported.
|
||||
1. ``--source PATH`` — explicit manual directories (repeatable) always
|
||||
win; git and local sources are ignored when this flag is used.
|
||||
2. The effective sources — the admin-managed ``git_sources`` table rows
|
||||
(both kinds), else the ``BOR_GIT_SOURCES`` (comma-separated) git-only
|
||||
fallback (:func:`app.rag.git_sources.effective_sources`, the same
|
||||
shared resolver the in-app Sync button uses). Git rows are cloned
|
||||
(first run, shallow ``--depth 1``) or fast-forwarded
|
||||
(``git pull --ff-only``) into ``BOR_SOURCES_DIR/<repo-name>/``
|
||||
(default ``~/bor-sources``); local rows are the existing directories
|
||||
themselves, walked directly. A failing clone/pull — or a local
|
||||
directory that is missing at run time — aborts the whole run
|
||||
*before* anything is imported.
|
||||
3. Fallback — the legacy ``DEFAULT_SOURCES`` (``~/Homelab`` +
|
||||
``~/Deployments``), kept for backwards compatibility.
|
||||
``~/Deployments``), kept for backwards compatibility (reached only
|
||||
while both the table and ``BOR_GIT_SOURCES`` are empty).
|
||||
|
||||
Imported formats (PLAN anchor A9, revised): ``md, markdown, txt, yaml,
|
||||
yml, json, py`` (case-insensitive; narrow with ``BOR_IMPORT_EXTENSIONS``).
|
||||
@@ -56,7 +60,7 @@ from app.core.debugging import configure_debugging
|
||||
from app.core.logging import configure_logging
|
||||
from app.db import SessionLocal
|
||||
from app.models import KbOverview
|
||||
from app.rag.git_sources import effective_git_sources
|
||||
from app.rag.git_sources import effective_sources
|
||||
from app.rag.importer import ImportSummary, import_sources
|
||||
from app.rag.llm import LLMClient
|
||||
from app.rag.overview import regenerate_overview
|
||||
@@ -78,8 +82,9 @@ def build_parser() -> argparse.ArgumentParser:
|
||||
type=Path,
|
||||
metavar="PATH",
|
||||
help=(
|
||||
"directory to import (repeatable; always wins over BOR_GIT_SOURCES; "
|
||||
"default when neither is given: ~/Homelab ~/Deployments)"
|
||||
"directory to import (repeatable; always wins over the git_sources "
|
||||
"DB rows and BOR_GIT_SOURCES; default when neither is given: "
|
||||
"~/Homelab ~/Deployments)"
|
||||
),
|
||||
)
|
||||
p.add_argument(
|
||||
@@ -115,31 +120,53 @@ def repo_name(url: str) -> str:
|
||||
|
||||
|
||||
def _resolve_sources(cli_sources: list[Path] | None, settings: Settings) -> list[Path]:
|
||||
"""Resolve the directories to import (phase 28, extended in phase 35).
|
||||
"""Resolve the directories to import (phase 28, extended in phases
|
||||
35 and 38).
|
||||
|
||||
Precedence: ``--source`` (explicit manual paths — always wins) >
|
||||
the effective git sources — the ``git_sources`` DB rows, else the
|
||||
``BOR_GIT_SOURCES`` fallback
|
||||
(:func:`app.rag.git_sources.effective_git_sources`; the import needs
|
||||
the database anyway, so resolution opens a short session and there
|
||||
is no DB-down branch) — each URL cloned/pulled via
|
||||
the effective sources — the ``git_sources`` DB rows (git + local),
|
||||
else the ``BOR_GIT_SOURCES`` git-only fallback
|
||||
(:func:`app.rag.git_sources.effective_sources`; the import needs the
|
||||
database anyway, so resolution opens a short session and there is
|
||||
no DB-down branch) — git rows cloned/pulled via
|
||||
:func:`scripts.git_sync.clone_or_pull` into
|
||||
``BOR_SOURCES_DIR/<repo-name>/`` > the legacy ``DEFAULT_SOURCES``.
|
||||
``BOR_SOURCES_DIR/<repo-name>/``, local rows walked directly (the
|
||||
stored directory, re-verified ``.is_dir()`` at run time) > the
|
||||
legacy ``DEFAULT_SOURCES``.
|
||||
|
||||
A :class:`GitSyncError` from a failing clone/pull propagates to
|
||||
A :class:`GitSyncError` from a failing clone/pull — or a missing
|
||||
local directory (``local source missing: <path>``) — propagates to
|
||||
:func:`main`, which aborts the run before importing anything.
|
||||
"""
|
||||
if cli_sources:
|
||||
return [path.expanduser() for path in cli_sources]
|
||||
db = SessionLocal()
|
||||
try:
|
||||
git_urls, origin = effective_git_sources(db)
|
||||
rows, origin = effective_sources(db)
|
||||
finally:
|
||||
db.close()
|
||||
if git_urls:
|
||||
logger.info("git sources: %d repo(s) origin=%s", len(git_urls), origin)
|
||||
if rows:
|
||||
git_count = sum(1 for row in rows if row.kind == "git")
|
||||
logger.info(
|
||||
"sources: %d repo(s) git=%d local=%d origin=%s",
|
||||
len(rows), git_count, len(rows) - git_count, origin,
|
||||
)
|
||||
sources_root = Path(settings.sources_dir).expanduser()
|
||||
return [clone_or_pull(url, sources_root / repo_name(url)) for url in git_urls]
|
||||
sources: list[Path] = []
|
||||
for row in rows:
|
||||
if row.kind == "git":
|
||||
sources.append(clone_or_pull(row.url, sources_root / repo_name(row.url)))
|
||||
else:
|
||||
# kind=local — the stored expanded path (phase 38 also
|
||||
# mirrors it in the NOT-NULL ``url`` location column, the
|
||||
# ``or`` keeps the type checker honest); a missing
|
||||
# directory aborts before importing, the same pre-import
|
||||
# fail-loud as a failing git clone.
|
||||
path = Path(row.path or row.url).expanduser()
|
||||
if not path.is_dir():
|
||||
raise GitSyncError(f"local source missing: {path}")
|
||||
sources.append(path)
|
||||
return sources
|
||||
return [path.expanduser() for path in DEFAULT_SOURCES]
|
||||
|
||||
|
||||
@@ -167,7 +194,7 @@ def main(argv: list[str] | None = None) -> int:
|
||||
try:
|
||||
sources = _resolve_sources(args.source, settings)
|
||||
except GitSyncError as e:
|
||||
print(f"import_docs: git sync failed: {e}", file=sys.stderr)
|
||||
print(f"import_docs: source sync failed: {e}", file=sys.stderr)
|
||||
return 1
|
||||
|
||||
logger.info(
|
||||
|
||||
@@ -0,0 +1,496 @@
|
||||
"""Phase 38 story E2E (Playwright): local directory sources.
|
||||
|
||||
Story: ``.agent/user_stories/local-directory-sources.md``
|
||||
Run in isolation (DB must be up: ``podman compose up -d db``):
|
||||
|
||||
uv run pytest tests/e2e/test_local_directory_sources.py -v --no-cov
|
||||
|
||||
The story gate for the **local directory** kind of the admin-managed
|
||||
source registry (phase 38): the admin adds an existing, non-git
|
||||
directory on the same admin page as the git repos (phase 35), and the
|
||||
real Sync button (phase 32) imports it — with add-time fail-loud
|
||||
validation (a missing/relative path is rejected inline, naming the
|
||||
path) and union pruning (a file deleted from the directory leaves the
|
||||
index on the next sync; removing the row stops it being a source).
|
||||
|
||||
The fixture is a **host temp dir** (``tmp_path_factory`` — the app
|
||||
server runs on the same host, so the path is visible to it) containing
|
||||
one plain ``.md`` with a distinctive sentinel token. No git anywhere in
|
||||
this suite (the directory is deliberately NOT a git repo — that is the
|
||||
point of the story), so no ``BOR_GIT_SOURCES`` and no clone: the sync
|
||||
pipeline under test is the ``kind=local`` branch (direct directory
|
||||
walk, re-verified ``.is_dir()`` at sync time) with prune over the union
|
||||
(the KB was truncated, so the fixture file is the only thing the sync
|
||||
can import — and the only thing it can prune).
|
||||
|
||||
Per-module app env (the conftest pattern, module-scoped — as in
|
||||
``test_sync_button.py``): this story's app boots WITHOUT
|
||||
``BOR_GIT_SOURCES`` (the env fallback is git-only by the phase's locked
|
||||
decision — local directories are DB-registered, no env var), so an
|
||||
empty table means "no sources configured" until the admin adds the
|
||||
directory through the real page.
|
||||
|
||||
Contract under test:
|
||||
|
||||
* anonymous: the sign-in gate (the phase-16/35 ``#git-sources-gate``
|
||||
pattern), the manager hidden (list + BOTH add forms inert), NO
|
||||
``/api/git-sources`` call, and 403 on the source routes + the sync
|
||||
trigger (the phase-35 regression assertions, A10);
|
||||
* admin: a missing path (``/nonexistent/bor-e2e``) 422s inline naming
|
||||
the path with no row added and the button never stale; the host temp
|
||||
dir adds (201 → row with the **Local** badge + the full path in a
|
||||
mono cell, input cleared, button re-enabled); the same path again
|
||||
409s inline ("already exists", path named) with no second row;
|
||||
* admin: the header **Sync** button (the phase-32 lifecycle, "Syncing…"
|
||||
→ "Synced HH:MM") imports the fixture file — it appears in
|
||||
``GET /api/docs`` (and its sentinel is in ``GET
|
||||
/api/documents/content``); deleting the file and syncing again prunes
|
||||
it (``pruned: 1``, gone from ``GET /api/docs`` — union prune); then
|
||||
removing the row on the page makes it disappear (accept the confirm;
|
||||
the empty state returns);
|
||||
* the new local form's a11y basics (UI Structure Check, AGENTS.md rule
|
||||
5): labeled input, role=alert error line, ≥44px target, 3px
|
||||
focus-visible outline.
|
||||
|
||||
Test → story mapping (Playwright Mapping Rule):
|
||||
1. ``test_anonymous_soft_gate_and_403s``
|
||||
2. ``test_admin_add_missing_path_then_dir_then_duplicate``
|
||||
3. ``test_admin_sync_imports_fixture_prunes_after_delete_removes_row``
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import os
|
||||
import re
|
||||
import subprocess
|
||||
import sys
|
||||
import time
|
||||
from collections.abc import Iterator
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
import pytest
|
||||
from playwright.sync_api import Dialog, Page, expect
|
||||
from sqlalchemy import text
|
||||
|
||||
from app.db import SessionLocal
|
||||
from e2e.auth_helpers import login
|
||||
from e2e.conftest import (
|
||||
ADMIN_PASSWORD,
|
||||
APP_PORT,
|
||||
SESSION_SECRET,
|
||||
USE_REAL_LLM,
|
||||
_wait_http,
|
||||
)
|
||||
|
||||
REPO = Path(__file__).resolve().parents[2]
|
||||
APP_URL = f"http://127.0.0.1:{APP_PORT}"
|
||||
|
||||
GIT_SOURCES_URL = "/git-sources.html"
|
||||
|
||||
#: The unique sentinel inside the fixture doc — its import into the KB
|
||||
#: (visible via GET /api/docs + /api/documents/content) proves the real
|
||||
#: sync walked the local directory.
|
||||
SENTINEL = "RESE-LOCAL-DIR-TOKEN-4d7e"
|
||||
FIXTURE_REL = "notes/bor-local-fixture.md"
|
||||
|
||||
#: A path that must NOT exist on the host — the add-time 422 subject.
|
||||
MISSING_PATH = "/nonexistent/bor-e2e"
|
||||
|
||||
#: "Synced HH:MM" — the local-time last-result label (header.js's
|
||||
#: fmtSyncTime), any hour/minute.
|
||||
SYNCED_LABEL = re.compile(r"Synced \d{1,2}:\d{2}")
|
||||
|
||||
#: A single-file import against the mock LLM is fast, but the sync runs
|
||||
#: the full pipeline (verify → walk → embed → overview) — same generous
|
||||
#: budget as test_sync_button.py, no client-side hard timeout.
|
||||
SYNC_TIMEOUT_MS = 60_000
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Fixtures
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
@pytest.fixture(scope="module")
|
||||
def local_dir(tmp_path_factory: pytest.TempPathFactory) -> Path:
|
||||
"""The story's ``~/Notes``: a plain (non-git) directory the admin
|
||||
registers as a source. Built under ``tmp_path_factory`` (module
|
||||
lifetime, like ``test_sync_button.py``'s fixture repo) and holding
|
||||
one A9-format fixture doc with the sentinel token. The app server
|
||||
runs on the same host, so this path is visible to it."""
|
||||
root = tmp_path_factory.mktemp("bor_local_dir") / "notes-dir"
|
||||
(root / "notes").mkdir(parents=True)
|
||||
(root / FIXTURE_REL).write_text(
|
||||
"# Local directory fixture\n"
|
||||
"\n"
|
||||
"One small note that exists only to prove the local-directory\n"
|
||||
"source story end to end: the admin adds this directory on the\n"
|
||||
"git sources page, the real Sync button walks it and imports it,\n"
|
||||
"and deleting the file + syncing again prunes it (union prune).\n"
|
||||
"\n"
|
||||
f"Marker: {SENTINEL}\n",
|
||||
encoding="utf-8",
|
||||
)
|
||||
assert (root / FIXTURE_REL).is_file()
|
||||
assert not (root / ".git").exists() # the story: NOT a git repo
|
||||
return root
|
||||
|
||||
|
||||
@pytest.fixture(scope="module")
|
||||
def app_server(mock_llm: int, local_dir: Path) -> Iterator[str]:
|
||||
"""The real app under test — per-module env: NO ``BOR_GIT_SOURCES``
|
||||
(the env fallback is git-only; local directories are DB-registered)
|
||||
and a scratch ``BOR_SOURCES_DIR`` (no git row ever syncs here, it is
|
||||
set for hygiene). The session app is never started in this isolated
|
||||
run, so no port clash."""
|
||||
env = dict(os.environ)
|
||||
env.pop("DEBUGPY", None)
|
||||
env["BOR_ENVIRONMENT"] = "e2e"
|
||||
env["BOR_STATIC_DIR"] = str(REPO / "frontend")
|
||||
env["BOR_LLM_BASE_URL"] = (
|
||||
"https://aipi.reeseapps.com/v1" if USE_REAL_LLM else f"http://127.0.0.1:{mock_llm}/v1"
|
||||
)
|
||||
# Mock-calibrated threshold (conftest pattern) — no chat turn is ever
|
||||
# sent in this suite, but the app boots with the same env shape.
|
||||
env["BOR_RELEVANCE_THRESHOLD"] = "0.30"
|
||||
env.setdefault(
|
||||
"BOR_DATABASE_URL",
|
||||
"postgresql+psycopg://reese:reese@localhost:5432/brain_of_reese",
|
||||
)
|
||||
# Phase 16: admin auth must be set or create_app() refuses to boot.
|
||||
env["BOR_ADMIN_PASSWORD"] = ADMIN_PASSWORD
|
||||
env["BOR_SESSION_SECRET"] = SESSION_SECRET
|
||||
# The repo's .env file carries the owner's BOR_GIT_SOURCES (the app
|
||||
# reads it from cwd) — override it with an EMPTY value (the env var
|
||||
# beats the .env file): the story is local-kind only, the env
|
||||
# fallback stays git-only, and an empty table + empty fallback must
|
||||
# mean "no sources configured" until the admin adds the directory.
|
||||
env["BOR_GIT_SOURCES"] = ""
|
||||
env["BOR_SOURCES_DIR"] = str(local_dir.parent / "checkouts")
|
||||
proc = subprocess.Popen(
|
||||
[sys.executable, "-m", "uvicorn", "app.main:app",
|
||||
"--host", "127.0.0.1", "--port", str(APP_PORT), "--log-level", "warning"],
|
||||
cwd=REPO,
|
||||
env=env,
|
||||
)
|
||||
try:
|
||||
_wait_http(f"{APP_URL}/api/health")
|
||||
yield APP_URL
|
||||
finally:
|
||||
proc.terminate()
|
||||
try:
|
||||
proc.wait(timeout=10)
|
||||
except subprocess.TimeoutExpired:
|
||||
proc.kill()
|
||||
|
||||
|
||||
@pytest.fixture(scope="module")
|
||||
def app_url(app_server: str) -> str:
|
||||
return app_server
|
||||
|
||||
|
||||
def _truncate_all() -> None:
|
||||
"""Fresh registry + KB per test (the E2E isolation pattern): the
|
||||
sync's counts and every GET /api/docs assertion must be this test's
|
||||
own doing. The E2E suites share one Postgres, and a leftover
|
||||
git_sources row would flip the sync from "no sources configured" to
|
||||
importing another suite's source (or a leftover document would show
|
||||
up in the docs list the pruned-union assertions inspect)."""
|
||||
with SessionLocal() as db:
|
||||
db.execute(text("TRUNCATE chunks, documents, query_log, kb_overview, git_sources"))
|
||||
db.commit()
|
||||
|
||||
|
||||
@pytest.fixture(autouse=True)
|
||||
def _clean(db_ready: None) -> Iterator[None]:
|
||||
_truncate_all()
|
||||
yield
|
||||
_truncate_all()
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Helpers
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _admin_git_sources_page(page: Page, app_url: str) -> None:
|
||||
"""Real form login landing on the sources page (admin settled:
|
||||
Sign out visible, the manager revealed by the page module)."""
|
||||
login(page, app_url, next=GIT_SOURCES_URL)
|
||||
expect(page).to_have_url(app_url + GIT_SOURCES_URL, timeout=30_000)
|
||||
expect(page.locator("#sign-out-btn")).to_be_visible(timeout=15_000)
|
||||
expect(page.locator("#git-sources-gate")).to_be_hidden()
|
||||
expect(page.locator("#git-sources-content")).to_be_visible()
|
||||
|
||||
|
||||
def _add_local_dir(page: Page, path: str) -> None:
|
||||
"""Add a local directory through the real page form and wait for the
|
||||
new row (the 201 → reload → row lifecycle of git-sources.js)."""
|
||||
page.fill("#local-source-path", path)
|
||||
page.click("#local-source-add")
|
||||
expect(
|
||||
page.locator("#git-sources-tbody tr", has_text=path)
|
||||
).to_have_count(1, timeout=30_000)
|
||||
|
||||
|
||||
def _click_sync(page: Page) -> None:
|
||||
"""The phase-32 button lifecycle: click → disabled + "Syncing…" →
|
||||
"Synced HH:MM" (re-enabled — never stale). The server status poll
|
||||
underneath is what the 2 s UI loop observes."""
|
||||
btn = page.locator("#sync-btn")
|
||||
expect(btn).to_be_visible()
|
||||
btn.click()
|
||||
expect(btn).to_be_disabled()
|
||||
expect(page.locator("#sync-label")).to_have_text("Syncing…")
|
||||
expect(page.locator("#sync-label")).to_have_text(SYNCED_LABEL, timeout=SYNC_TIMEOUT_MS)
|
||||
expect(btn).to_be_enabled()
|
||||
|
||||
|
||||
def _wait_sync_done(page: Page, app_url: str, timeout_s: float = 60.0) -> dict[str, Any]:
|
||||
"""Poll the (cookie-authenticated) status endpoint until the run
|
||||
reaches a terminal state — exactly what the UI's 2 s poll loop
|
||||
observes (test_sync_button.py's helper)."""
|
||||
deadline = time.monotonic() + timeout_s
|
||||
body: dict[str, Any] = {}
|
||||
while time.monotonic() < deadline:
|
||||
r = page.request.get(f"{app_url}/api/sync/status")
|
||||
assert r.status == 200
|
||||
body = r.json()
|
||||
if body["state"] in ("success", "failed"):
|
||||
return body
|
||||
time.sleep(0.5)
|
||||
raise AssertionError(f"sync did not reach a terminal state: {body}")
|
||||
|
||||
|
||||
def _docs(page: Page, app_url: str) -> list[dict[str, Any]]:
|
||||
"""GET /api/docs as the signed-in page (admin cookie rides along)."""
|
||||
r = page.request.get(f"{app_url}/api/docs")
|
||||
assert r.status == 200, r.text
|
||||
return r.json()["documents"]
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 1. Anonymous: the soft gate, inert manager, no API calls, 403s
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_anonymous_soft_gate_and_403s(
|
||||
page: Page, app_url: str, db_ready: None
|
||||
) -> None:
|
||||
"""The phase-16/35 gate on this page (regression through the phase-38
|
||||
form): anonymous visitors see the sign-in gate and a fully hidden
|
||||
manager (list + git form + local form), the page never calls the
|
||||
admin API, and every admin route 403s (A10)."""
|
||||
page.set_default_timeout(30_000)
|
||||
|
||||
api_calls: list[str] = []
|
||||
page.on(
|
||||
"request",
|
||||
lambda r: api_calls.append(r.url)
|
||||
if "/api/git-sources" in r.url
|
||||
else None,
|
||||
)
|
||||
|
||||
page.goto(app_url + GIT_SOURCES_URL)
|
||||
expect(page.locator("#sign-in-link")).to_be_visible(timeout=15_000)
|
||||
expect(page.locator("#sign-out-btn")).to_be_hidden()
|
||||
|
||||
gate = page.locator("#git-sources-gate")
|
||||
expect(gate).to_be_visible()
|
||||
expect(gate).to_contain_text("Sign in to manage the git sources")
|
||||
|
||||
# The manager is absent/inert: list, BOTH add forms, env note — all
|
||||
# inside the hidden #git-sources-content.
|
||||
expect(page.locator("#git-sources-content")).to_be_hidden()
|
||||
expect(page.locator("#git-sources-table")).to_be_hidden()
|
||||
expect(page.locator("#git-source-form")).to_be_hidden()
|
||||
expect(page.locator("#local-source-form")).to_be_hidden()
|
||||
expect(page.locator("#git-sources-env-note")).to_be_hidden()
|
||||
|
||||
# The gate never called the admin API…
|
||||
assert api_calls == [], f"anonymous page called the git sources API: {api_calls}"
|
||||
# …and the API 403s anonymous callers (the phase-35 assertions):
|
||||
# all three source routes, for BOTH kinds, plus the sync trigger.
|
||||
assert page.request.get(f"{app_url}/api/git-sources").status == 403
|
||||
assert (
|
||||
page.request.post(
|
||||
f"{app_url}/api/git-sources", data={"url": "https://example.com/x.git"}
|
||||
).status
|
||||
== 403
|
||||
)
|
||||
assert (
|
||||
page.request.post(
|
||||
f"{app_url}/api/git-sources", data={"kind": "local", "path": "/tmp"}
|
||||
).status
|
||||
== 403
|
||||
)
|
||||
assert (
|
||||
page.request.delete(
|
||||
f"{app_url}/api/git-sources/00000000-0000-0000-0000-000000000000"
|
||||
).status
|
||||
== 403
|
||||
)
|
||||
assert page.request.post(f"{app_url}/api/sync").status == 403
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 2. Admin: add validation (missing path, dir, duplicate) + local form
|
||||
# a11y basics
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_admin_add_missing_path_then_dir_then_duplicate(
|
||||
page: Page, app_url: str, local_dir: Path, db_ready: None
|
||||
) -> None:
|
||||
"""Add-time fail-loud validation on the real page: a missing path
|
||||
422s inline NAMING the path (no row, never-stale button, the input
|
||||
survives for one edit); the host temp dir adds (row with the Local
|
||||
badge + full path, input cleared); the same path again 409s inline
|
||||
("already exists", path named, no second row)."""
|
||||
page.set_default_timeout(30_000)
|
||||
_admin_git_sources_page(page, app_url)
|
||||
expect(page.locator("#git-sources-tbody tr")).to_have_count(0)
|
||||
|
||||
error = page.locator("#local-source-error")
|
||||
add_btn = page.locator("#local-source-add")
|
||||
|
||||
# --- missing path: inline 422 naming it, NO row, button recovers ---
|
||||
page.fill("#local-source-path", MISSING_PATH)
|
||||
add_btn.click()
|
||||
expect(error).to_be_visible(timeout=30_000)
|
||||
assert error.get_attribute("role") == "alert"
|
||||
expect(error).to_contain_text(MISSING_PATH)
|
||||
expect(error).to_contain_text("not a directory")
|
||||
expect(page.locator("#git-sources-tbody tr")).to_have_count(0)
|
||||
expect(add_btn).to_be_enabled()
|
||||
expect(add_btn).to_have_text("Add directory")
|
||||
expect(page.locator("#local-source-path")).to_have_value(MISSING_PATH)
|
||||
|
||||
# --- the temp dir: 201 → the row appears with the Local badge ------
|
||||
_add_local_dir(page, str(local_dir))
|
||||
row = page.locator("#git-sources-tbody tr", has_text=str(local_dir))
|
||||
expect(row).to_have_count(1)
|
||||
badge = row.locator("span.git-source-kind")
|
||||
expect(badge).to_have_text("Local")
|
||||
expect(badge).to_have_class(re.compile(r"\bis-local\b"))
|
||||
# The mono cell carries the full path (rendered as text)…
|
||||
expect(row.locator("td.git-source-url-cell code")).to_have_text(str(local_dir))
|
||||
# …and the row's Remove button is labeled with the kind + path.
|
||||
expect(row.locator(".git-source-remove")).to_have_attribute(
|
||||
"aria-label", f"Remove local source: {local_dir}"
|
||||
)
|
||||
# The 201 cleared the input and re-enabled the button (never stale).
|
||||
expect(page.locator("#local-source-path")).to_have_value("")
|
||||
expect(add_btn).to_be_enabled()
|
||||
expect(add_btn).to_have_text("Add directory")
|
||||
# The API agrees: kind=local with the stored (expanded) path.
|
||||
r = page.request.get(f"{app_url}/api/git-sources")
|
||||
assert r.status == 200, r.text
|
||||
body = r.json()
|
||||
assert body["from_env"] is False
|
||||
assert [
|
||||
(s["kind"], s["path"]) for s in body["sources"]
|
||||
] == [("local", str(local_dir))]
|
||||
|
||||
# --- duplicate: inline 409 naming the path, NO second row -----------
|
||||
page.fill("#local-source-path", str(local_dir))
|
||||
add_btn.click()
|
||||
expect(error).to_be_visible(timeout=30_000)
|
||||
expect(error).to_contain_text("already exists")
|
||||
expect(error).to_contain_text(str(local_dir))
|
||||
expect(page.locator("#git-sources-tbody tr")).to_have_count(1)
|
||||
expect(add_btn).to_be_enabled()
|
||||
expect(add_btn).to_have_text("Add directory")
|
||||
expect(page.locator("#local-source-path")).to_have_value(str(local_dir))
|
||||
|
||||
# --- the new form's a11y basics (UI Structure Check, AGENTS.md 5) ---
|
||||
expect(page.get_by_label("Add a local directory")).to_have_count(1)
|
||||
box = add_btn.bounding_box()
|
||||
assert box is not None and box["height"] >= 44, f"target too small: {box}"
|
||||
page.focus("#local-source-path")
|
||||
outline = page.evaluate(
|
||||
"() => getComputedStyle(document.querySelector('#local-source-path')).outlineWidth"
|
||||
)
|
||||
assert outline == "3px", f"focus-visible outline missing: {outline!r}"
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 3. Admin: the real Sync imports the fixture; deleting the file +
|
||||
# syncing again prunes it (union prune); removing the row ends it
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_admin_sync_imports_fixture_prunes_after_delete_removes_row(
|
||||
page: Page, app_url: str, local_dir: Path, db_ready: None
|
||||
) -> None:
|
||||
"""The phase-32 button drives the phase-38 pipeline: the header Sync
|
||||
imports the local directory's fixture file (GET /api/docs shows it,
|
||||
the sentinel is in its content); deleting the file and syncing again
|
||||
prunes it (``pruned: 1`` — prune over the union); then removing the
|
||||
row on the page makes it disappear (the empty state returns)."""
|
||||
page.set_default_timeout(30_000)
|
||||
_admin_git_sources_page(page, app_url)
|
||||
|
||||
# Fresh registry (the autouse fixture truncated it) — add the source
|
||||
# through the real page, then run the sync lifecycle against it.
|
||||
_add_local_dir(page, str(local_dir))
|
||||
|
||||
# --- run 1: the real sync walks the local dir and imports the file -
|
||||
_click_sync(page)
|
||||
body = _wait_sync_done(page, app_url)
|
||||
assert body["state"] == "success", body
|
||||
assert body["detail"]["added"] == 1, body["detail"]
|
||||
assert body["detail"]["pruned"] == 0, body["detail"]
|
||||
|
||||
# The fixture doc is in GET /api/docs…
|
||||
docs = _docs(page, app_url)
|
||||
fixture_docs = [d for d in docs if d["path"] == FIXTURE_REL]
|
||||
assert len(fixture_docs) == 1, f"fixture doc missing from /api/docs: {docs}"
|
||||
assert fixture_docs[0]["source"] == local_dir.name
|
||||
# …and its content carries the sentinel (the walk imported THIS file).
|
||||
content = page.request.get(
|
||||
f"{app_url}/api/documents/content"
|
||||
f"?source={local_dir.name}&path={FIXTURE_REL}"
|
||||
)
|
||||
assert content.status == 200, content.text
|
||||
assert SENTINEL in content.json()["content"]
|
||||
|
||||
# --- run 2: file deleted → the next sync prunes it (union prune) ---
|
||||
(local_dir / FIXTURE_REL).unlink()
|
||||
_click_sync(page)
|
||||
body = _wait_sync_done(page, app_url)
|
||||
assert body["state"] == "success", body
|
||||
assert body["detail"]["pruned"] == 1, body["detail"]
|
||||
assert body["detail"]["added"] == 0, body["detail"]
|
||||
docs = _docs(page, app_url)
|
||||
assert [d for d in docs if d["path"] == FIXTURE_REL] == [], (
|
||||
f"fixture doc survived the prune: {docs}"
|
||||
)
|
||||
|
||||
# --- remove the row: accept the confirm → it disappears ------------
|
||||
removes: list[str] = []
|
||||
page.on(
|
||||
"request",
|
||||
lambda r: removes.append(r.url)
|
||||
if r.method == "DELETE" and "/api/git-sources/" in r.url
|
||||
else None,
|
||||
)
|
||||
|
||||
def handle_dialog(dialog: Dialog) -> None:
|
||||
# "Remove this local source…? Its documents stay indexed until
|
||||
# the next sync prunes them." — accept it.
|
||||
dialog.accept()
|
||||
|
||||
page.on("dialog", handle_dialog)
|
||||
try:
|
||||
row = page.locator("#git-sources-tbody tr", has_text=str(local_dir))
|
||||
row.locator(".git-source-remove").click()
|
||||
expect(page.locator("#git-sources-tbody tr")).to_have_count(0, timeout=30_000)
|
||||
expect(page.locator("#git-sources-empty")).to_be_visible()
|
||||
finally:
|
||||
page.remove_listener("dialog", handle_dialog)
|
||||
assert len(removes) == 1, f"expected one DELETE, saw: {removes}"
|
||||
# The registry is empty again — and with no env git list, a further
|
||||
# sync would fail loudly ("no sources configured (git or local)").
|
||||
r = page.request.get(f"{app_url}/api/git-sources")
|
||||
assert r.json() == {"sources": [], "from_env": True}
|
||||
@@ -1,4 +1,5 @@
|
||||
"""Integration: the admin git-sources CRUD API (phase 35, task 02).
|
||||
"""Integration: the admin sources CRUD API (phase 35, task 02; local
|
||||
kind, phase 38, task 02).
|
||||
|
||||
Real Postgres (``podman compose up -d db``); the ``BOR_GIT_SOURCES``
|
||||
fallback is exercised deterministically by monkeypatching the router's
|
||||
@@ -7,17 +8,25 @@ fallback is exercised deterministically by monkeypatching the router's
|
||||
|
||||
Contract under test:
|
||||
|
||||
* anonymous → 403 ``{"detail": "admin only"}`` on GET, POST, and DELETE
|
||||
(phase 16 pattern, same as ``/api/sync``);
|
||||
* GET — empty table + env set → the env rows with ``from_env: true`` and
|
||||
null ``id``/``added_at``; empty table + empty env → ``sources: []``
|
||||
with ``from_env: true``; any DB rows → ``from_env: false`` and the env
|
||||
var is ignored (the phase's locked decision); DB rows ordered by
|
||||
``(added_at, id)``;
|
||||
* POST — 201 stored trimmed; duplicate (even with different surrounding
|
||||
whitespace) → 409 with a generic detail that never echoes the URL
|
||||
(credential safety), including when only the DB unique index catches
|
||||
it; bad shape / blank / >500 chars → 422, also input-free;
|
||||
* anonymous → 403 ``{"detail": "admin only"}`` on GET, POST (git and
|
||||
local), and DELETE (phase 16 pattern, same as ``/api/sync``);
|
||||
* GET — rows carry ``kind`` + ``path`` (phase 38); empty table + env
|
||||
set → the git-only env rows (``kind: "git"``, ``path: null``) with
|
||||
``from_env: true`` and null ``id``/``added_at``; empty table + empty
|
||||
env → ``sources: []`` with ``from_env: true``; any DB rows →
|
||||
``from_env: false`` and the env var is ignored (the phase's locked
|
||||
decision); DB rows ordered by ``(added_at, id)``;
|
||||
* POST ``kind=git`` (default) — 201 stored trimmed; duplicate (even with
|
||||
different surrounding whitespace) → 409 with a generic detail that
|
||||
never echoes the URL (credential safety), including when only the DB
|
||||
unique index catches it; bad shape / blank / >500 chars → 422, also
|
||||
input-free (the phase-35 contract, unchanged);
|
||||
* POST ``kind=local`` (phase 38) — existing directory → 201, stored row
|
||||
carries ``kind=local`` + the path expanded (``~`` resolved) and
|
||||
trimmed; relative / missing / not-a-directory path → 422 naming the
|
||||
path (not a secret); duplicate path → 409 naming the path (unique
|
||||
index as backstop); wrong field combinations (git without url, local
|
||||
without path, both kinds' fields) → 422;
|
||||
* DELETE — 204 and gone; an emptied table falls back to the env list
|
||||
again; unknown id → 404.
|
||||
|
||||
@@ -28,6 +37,7 @@ from __future__ import annotations
|
||||
import uuid
|
||||
from collections.abc import Iterator
|
||||
from datetime import UTC, datetime, timedelta
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
import pytest
|
||||
@@ -66,6 +76,10 @@ def test_anonymous_gets_403_on_all_routes(client: TestClient, db: Session) -> No
|
||||
r = client.post("/api/git-sources", json={"url": "https://anon.example.com/x.git"})
|
||||
assert r.status_code == 403
|
||||
assert r.json() == {"detail": "admin only"}
|
||||
# The phase-38 local kind is gated the same way.
|
||||
r = client.post("/api/git-sources", json={"kind": "local", "path": "/tmp"})
|
||||
assert r.status_code == 403
|
||||
assert r.json() == {"detail": "admin only"}
|
||||
r = client.delete(f"/api/git-sources/{uuid.uuid4()}")
|
||||
assert r.status_code == 403
|
||||
assert r.json() == {"detail": "admin only"}
|
||||
@@ -89,10 +103,23 @@ def test_get_empty_table_with_env_returns_env_rows(
|
||||
assert r.status_code == 200
|
||||
body = r.json()
|
||||
assert body["from_env"] is True
|
||||
# Whitespace-trimmed, empty entries dropped, order preserved; null ids.
|
||||
# Whitespace-trimmed, empty entries dropped, order preserved; null
|
||||
# ids; the env fallback is git-only (phase 38: kind + path fields).
|
||||
assert body["sources"] == [
|
||||
{"id": None, "url": "https://a.example.com/one.git", "added_at": None},
|
||||
{"id": None, "url": "git@b.example.com:two.git", "added_at": None},
|
||||
{
|
||||
"id": None,
|
||||
"kind": "git",
|
||||
"url": "https://a.example.com/one.git",
|
||||
"path": None,
|
||||
"added_at": None,
|
||||
},
|
||||
{
|
||||
"id": None,
|
||||
"kind": "git",
|
||||
"url": "git@b.example.com:two.git",
|
||||
"path": None,
|
||||
"added_at": None,
|
||||
},
|
||||
]
|
||||
|
||||
|
||||
@@ -248,6 +275,200 @@ def test_post_rejects_blank_and_oversized_urls(admin_client: TestClient, db: Ses
|
||||
assert db.execute(text("SELECT count(*) FROM git_sources")).scalar_one() == 1
|
||||
|
||||
|
||||
# --- POST: local kind (phase 38, task 02) ----------------------------------
|
||||
|
||||
|
||||
def test_post_local_creates_stored_row_with_expanded_path(
|
||||
admin_client: TestClient,
|
||||
tmp_path: Path,
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
) -> None:
|
||||
"""``kind=local`` + an existing directory → 201; the stored row
|
||||
carries ``kind=local`` and the path expanded (``~`` resolved via the
|
||||
server's ``HOME``, whitespace trimmed)."""
|
||||
monkeypatch.setenv("HOME", str(tmp_path / "home"))
|
||||
real_dir = tmp_path / "home" / "notes"
|
||||
real_dir.mkdir(parents=True)
|
||||
|
||||
r = admin_client.post("/api/git-sources", json={"kind": "local", "path": " ~/notes\t"})
|
||||
assert r.status_code == 201, r.text
|
||||
body = r.json()
|
||||
# The phase-35 response shape is unchanged — the local row reports
|
||||
# its (expanded) path in ``url``; ``kind`` + ``path`` via GET.
|
||||
assert set(body) == {"id", "url", "added_at"}
|
||||
uuid.UUID(body["id"])
|
||||
assert body["url"] == str(real_dir)
|
||||
assert body["added_at"] is not None
|
||||
|
||||
row = admin_client.get("/api/git-sources").json()["sources"][0]
|
||||
assert row["kind"] == "local"
|
||||
assert row["path"] == str(real_dir)
|
||||
assert row["url"] == str(real_dir)
|
||||
assert row["id"] is not None
|
||||
assert row["added_at"] is not None
|
||||
|
||||
|
||||
def test_post_local_stores_trimmed_normalized_path(
|
||||
admin_client: TestClient, tmp_path: Path
|
||||
) -> None:
|
||||
"""Absolute path with surrounding whitespace + trailing slash →
|
||||
stored clean (trimmed, normalized)."""
|
||||
real_dir = tmp_path / "plain"
|
||||
real_dir.mkdir()
|
||||
|
||||
r = admin_client.post("/api/git-sources", json={"kind": "local", "path": f" {real_dir}/ "})
|
||||
assert r.status_code == 201, r.text
|
||||
row = admin_client.get("/api/git-sources").json()["sources"][0]
|
||||
assert row["path"] == str(real_dir)
|
||||
|
||||
|
||||
def test_post_local_relative_path_returns_422_naming_path(
|
||||
admin_client: TestClient, db: Session
|
||||
) -> None:
|
||||
"""A relative path fails loud at add-time — 422 naming the path
|
||||
(relative or not, it is never stored)."""
|
||||
r = admin_client.post("/api/git-sources", json={"kind": "local", "path": "relative/dir"})
|
||||
assert r.status_code == 422
|
||||
assert r.json()["detail"] == "local source path is not a directory: relative/dir"
|
||||
assert db.execute(text("SELECT count(*) FROM git_sources")).scalar_one() == 0
|
||||
|
||||
|
||||
def test_post_local_missing_path_returns_422_naming_path(
|
||||
admin_client: TestClient, db: Session
|
||||
) -> None:
|
||||
"""A missing (absolute) path is a user error → 422 naming the path so
|
||||
the owner sees exactly which directory failed."""
|
||||
missing = f"/nonexistent/bor-test-{uuid.uuid4()}"
|
||||
r = admin_client.post("/api/git-sources", json={"kind": "local", "path": missing})
|
||||
assert r.status_code == 422
|
||||
assert r.json()["detail"] == f"local source path is not a directory: {missing}"
|
||||
assert db.execute(text("SELECT count(*) FROM git_sources")).scalar_one() == 0
|
||||
|
||||
|
||||
def test_post_local_file_not_dir_returns_422(
|
||||
admin_client: TestClient, tmp_path: Path
|
||||
) -> None:
|
||||
"""An existing *file* is not a directory → 422 naming the path."""
|
||||
a_file = tmp_path / "a-file.md"
|
||||
a_file.write_text("not a directory")
|
||||
r = admin_client.post("/api/git-sources", json={"kind": "local", "path": str(a_file)})
|
||||
assert r.status_code == 422
|
||||
assert r.json()["detail"] == f"local source path is not a directory: {a_file}"
|
||||
|
||||
|
||||
def test_post_local_duplicate_path_returns_409_naming_path(
|
||||
admin_client: TestClient, tmp_path: Path
|
||||
) -> None:
|
||||
"""Duplicate path (even with different surrounding whitespace) → 409
|
||||
naming the path (a path is not a secret, unlike a git URL)."""
|
||||
real_dir = tmp_path / "dups"
|
||||
real_dir.mkdir()
|
||||
|
||||
assert admin_client.post("/api/git-sources", json={"kind": "local", "path": str(real_dir)})
|
||||
r = admin_client.post("/api/git-sources", json={"kind": "local", "path": f" {real_dir}\t"})
|
||||
assert r.status_code == 409
|
||||
detail = r.json()["detail"]
|
||||
assert detail == f"a local source with this path already exists: {real_dir}"
|
||||
# Exactly one row stored.
|
||||
assert len(admin_client.get("/api/git-sources").json()["sources"]) == 1
|
||||
|
||||
|
||||
def test_post_local_concurrent_insert_backstop_still_409(
|
||||
admin_client: TestClient, tmp_path: Path, monkeypatch: pytest.MonkeyPatch
|
||||
) -> None:
|
||||
"""If the duplicate pre-check misses (a concurrent insert lands
|
||||
between the check and the commit), the DB unique index on ``path``
|
||||
still yields the 409 naming the path — never a 500."""
|
||||
real_dir = tmp_path / "backstop"
|
||||
real_dir.mkdir()
|
||||
assert (
|
||||
admin_client.post("/api/git-sources", json={"kind": "local", "path": str(real_dir)})
|
||||
.status_code
|
||||
== 201
|
||||
)
|
||||
|
||||
real_select = git_sources_api.select
|
||||
|
||||
def blind_select(*args: Any, **kwargs: Any) -> Any:
|
||||
if args and args[0] is GitSource: # the duplicate pre-check
|
||||
# …now never matches — only the unique index can catch it.
|
||||
return real_select(GitSource).where(GitSource.url == "zz-never-matches")
|
||||
return real_select(*args, **kwargs)
|
||||
|
||||
monkeypatch.setattr(git_sources_api, "select", blind_select)
|
||||
r = admin_client.post("/api/git-sources", json={"kind": "local", "path": str(real_dir)})
|
||||
assert r.status_code == 409
|
||||
assert r.json()["detail"] == f"a local source with this path already exists: {real_dir}"
|
||||
|
||||
|
||||
def test_post_wrong_field_combinations_return_422(
|
||||
admin_client: TestClient, db: Session, tmp_path: Path
|
||||
) -> None:
|
||||
"""git without url, local without path, and both kinds' fields are
|
||||
422 with fixed details — nothing is stored."""
|
||||
real_dir = tmp_path / "combo"
|
||||
real_dir.mkdir()
|
||||
|
||||
assert admin_client.post("/api/git-sources", json={"kind": "git"}).status_code == 422
|
||||
r = admin_client.post(
|
||||
"/api/git-sources",
|
||||
json={"kind": "git", "url": "https://example.com/both.git", "path": str(real_dir)},
|
||||
)
|
||||
assert r.status_code == 422
|
||||
assert r.json()["detail"] == "a git source takes a url, not a path"
|
||||
assert admin_client.post("/api/git-sources", json={"kind": "local"}).status_code == 422
|
||||
# Whitespace-only path trims to empty → 422 as well (the schema's
|
||||
# min-length guard).
|
||||
assert (
|
||||
admin_client.post("/api/git-sources", json={"kind": "local", "path": " "}).status_code
|
||||
== 422
|
||||
)
|
||||
r = admin_client.post(
|
||||
"/api/git-sources",
|
||||
json={"kind": "local", "url": "https://example.com/both.git", "path": str(real_dir)},
|
||||
)
|
||||
assert r.status_code == 422
|
||||
assert r.json()["detail"] == "a local source takes a path, not a url"
|
||||
# Unknown kind and an oversized path are 422 too.
|
||||
assert (
|
||||
admin_client.post(
|
||||
"/api/git-sources", json={"kind": "svn", "url": "https://example.com/x.git"}
|
||||
).status_code
|
||||
== 422
|
||||
)
|
||||
assert (
|
||||
admin_client.post(
|
||||
"/api/git-sources", json={"kind": "local", "path": "/" + "x" * 2000}
|
||||
).status_code
|
||||
== 422
|
||||
)
|
||||
assert db.execute(text("SELECT count(*) FROM git_sources")).scalar_one() == 0
|
||||
|
||||
|
||||
def test_get_mixed_kinds_in_added_order(
|
||||
admin_client: TestClient, db: Session, tmp_path: Path
|
||||
) -> None:
|
||||
"""GET mixes git + local rows in ``(added_at, id)`` order; git rows
|
||||
report ``path: null``, local rows their stored path."""
|
||||
git_url = "https://example.com/mixed.git"
|
||||
db.add(GitSource(url=git_url, kind="git", added_at=datetime.now(UTC) - timedelta(hours=1)))
|
||||
db.commit()
|
||||
real_dir = tmp_path / "mixed"
|
||||
real_dir.mkdir()
|
||||
assert admin_client.post("/api/git-sources", json={"kind": "local", "path": str(real_dir)})
|
||||
|
||||
body = admin_client.get("/api/git-sources").json()
|
||||
assert body["from_env"] is False
|
||||
assert [s["url"] for s in body["sources"]] == [git_url, str(real_dir)]
|
||||
git_row, local_row = body["sources"]
|
||||
assert git_row["kind"] == "git"
|
||||
assert git_row["path"] is None
|
||||
assert git_row["id"] is not None
|
||||
assert local_row["kind"] == "local"
|
||||
assert local_row["path"] == str(real_dir)
|
||||
assert local_row["id"] is not None
|
||||
|
||||
|
||||
# --- DB rows win over env ---------------------------------------------------
|
||||
|
||||
|
||||
@@ -284,11 +505,18 @@ def test_delete_removes_row_and_falls_back_to_env(
|
||||
|
||||
assert admin_client.delete(f"/api/git-sources/{created.json()['id']}").status_code == 204
|
||||
|
||||
# The table is empty again → the env fallback is live once more.
|
||||
# The table is empty again → the env fallback is live once more
|
||||
# (git-only rows, phase 38).
|
||||
body = admin_client.get("/api/git-sources").json()
|
||||
assert body["from_env"] is True
|
||||
assert body["sources"] == [
|
||||
{"id": None, "url": "https://env.example.com/env.git", "added_at": None}
|
||||
{
|
||||
"id": None,
|
||||
"kind": "git",
|
||||
"url": "https://env.example.com/env.git",
|
||||
"path": None,
|
||||
"added_at": None,
|
||||
}
|
||||
]
|
||||
|
||||
|
||||
|
||||
@@ -1,28 +1,36 @@
|
||||
"""Integration test: ``import_docs`` git-source resolution (phase 28, task 03).
|
||||
"""Integration test: ``import_docs`` source resolution (phase 28, task
|
||||
03; phase 35 re-points at the shared resolver; phase 38 adds the local
|
||||
kind).
|
||||
|
||||
Drives ``scripts.import_docs`` end to end with a fake ``clone_or_pull`` (no
|
||||
real git, no network) and a recording fake ``import_sources`` (no real
|
||||
DB), covering:
|
||||
Drives ``scripts.import_docs`` end to end with a fake ``clone_or_pull``
|
||||
(no real git, no network) and a recording fake ``import_sources`` (no
|
||||
real DB), covering:
|
||||
|
||||
- Effective git sources set (phase 35: the shared resolver — stubbed
|
||||
here, keeping this file's no-real-DB style) → each URL is cloned/pulled
|
||||
into ``BOR_SOURCES_DIR/<repo-name>/`` and exactly those dirs are
|
||||
imported.
|
||||
- DB rows win over ``BOR_GIT_SOURCES`` (the resolver's ``db`` origin —
|
||||
the env list is ignored).
|
||||
- ``--source`` still wins over git sources (no git at all, no resolver
|
||||
- Effective sources set (phase 35: the shared resolver — stubbed here,
|
||||
keeping this file's no-real-DB style) → each git URL is cloned/pulled
|
||||
into ``BOR_SOURCES_DIR/<repo-name>/``; local rows are their existing
|
||||
directories, walked directly; exactly those dirs are imported.
|
||||
- DB rows (both kinds) win over ``BOR_GIT_SOURCES`` (the resolver's
|
||||
``db`` origin — the env list is ignored; the env fallback stays
|
||||
git-only).
|
||||
- ``--source`` still wins over the DB rows (no git at all, no resolver
|
||||
call).
|
||||
- No git sources + no ``--source`` → the legacy ``DEFAULT_SOURCES``.
|
||||
- No sources configured + no ``--source`` → the legacy
|
||||
``DEFAULT_SOURCES``.
|
||||
- A failing git sync → exit code 1, an error naming the failing repo on
|
||||
stderr, and **zero** import attempts.
|
||||
- A missing local directory → the same pre-import fail-loud: exit code
|
||||
1, ``local source missing: <path>`` on stderr, zero import attempts.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import re
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
|
||||
from app.config import Settings
|
||||
from app.models import GitSource
|
||||
from app.rag.importer import ImportSummary
|
||||
from scripts import import_docs
|
||||
from scripts.git_sync import GitSyncError
|
||||
@@ -33,6 +41,16 @@ def _settings(git_sources: str = "", sources_dir: str = "~/bor-sources") -> Sett
|
||||
return Settings(_env_file=None, git_sources=git_sources, sources_dir=sources_dir) # pyright: ignore[reportCallIssue]
|
||||
|
||||
|
||||
def _git_row(url: str) -> GitSource:
|
||||
return GitSource(url=url, kind="git")
|
||||
|
||||
|
||||
def _local_row(path: str) -> GitSource:
|
||||
"""A local row as the phase-38 API stores it: the expanded path in
|
||||
both ``path`` and the NOT-NULL ``url`` location column."""
|
||||
return GitSource(url=path, kind="local", path=path)
|
||||
|
||||
|
||||
class FakeImportSources:
|
||||
"""Records every ``import_sources`` call instead of touching a DB."""
|
||||
|
||||
@@ -97,11 +115,15 @@ def test_resolve_sources_git_urls_cloned_into_sources_dir(
|
||||
calls, fake = _fake_clone_factory()
|
||||
monkeypatch.setattr(import_docs, "clone_or_pull", fake)
|
||||
# Phase 35: resolution goes through the shared resolver (stubbed —
|
||||
# this file keeps its no-real-DB style); the URLs are the env list.
|
||||
# this file keeps its no-real-DB style); the rows are the env list,
|
||||
# surfaced as synthetic git rows.
|
||||
monkeypatch.setattr(
|
||||
import_docs,
|
||||
"effective_git_sources",
|
||||
lambda db: (["https://host/a/homelab.git", "git@host:user/deploy.git"], "env"),
|
||||
"effective_sources",
|
||||
lambda db: (
|
||||
[_git_row("https://host/a/homelab.git"), _git_row("git@host:user/deploy.git")],
|
||||
"env",
|
||||
),
|
||||
)
|
||||
settings = _settings(
|
||||
git_sources="https://host/a/homelab.git, git@host:user/deploy.git ,",
|
||||
@@ -140,8 +162,8 @@ def test_resolve_sources_db_rows_win_over_env(
|
||||
monkeypatch.setattr(import_docs, "clone_or_pull", fake)
|
||||
monkeypatch.setattr(
|
||||
import_docs,
|
||||
"effective_git_sources",
|
||||
lambda db: (["https://db.example/only.git"], "db"),
|
||||
"effective_sources",
|
||||
lambda db: ([_git_row("https://db.example/only.git")], "db"),
|
||||
)
|
||||
settings = _settings(
|
||||
git_sources="https://env.example/ignored.git",
|
||||
@@ -158,7 +180,7 @@ def test_resolve_sources_defaults_when_nothing_configured(
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
) -> None:
|
||||
# Both origins empty (the resolver's ``([], "env")``) → legacy dirs.
|
||||
monkeypatch.setattr(import_docs, "effective_git_sources", lambda db: ([], "env"))
|
||||
monkeypatch.setattr(import_docs, "effective_sources", lambda db: ([], "env"))
|
||||
sources = import_docs._resolve_sources(None, _settings())
|
||||
assert sources == [p.expanduser() for p in import_docs.DEFAULT_SOURCES]
|
||||
|
||||
@@ -178,8 +200,11 @@ def test_main_git_sources_clone_then_import(
|
||||
monkeypatch.setattr(import_docs, "get_settings", lambda: settings)
|
||||
monkeypatch.setattr(
|
||||
import_docs,
|
||||
"effective_git_sources",
|
||||
lambda db: (["https://host/a/homelab.git", "https://host/a/deploy.git"], "env"),
|
||||
"effective_sources",
|
||||
lambda db: (
|
||||
[_git_row("https://host/a/homelab.git"), _git_row("https://host/a/deploy.git")],
|
||||
"env",
|
||||
),
|
||||
)
|
||||
calls, fake = _fake_clone_factory()
|
||||
monkeypatch.setattr(import_docs, "clone_or_pull", fake)
|
||||
@@ -227,6 +252,55 @@ def test_main_cli_source_still_imports_manual_dir(
|
||||
assert fake_import.calls[0]["prune"] is False
|
||||
|
||||
|
||||
def test_resolve_sources_mixed_git_and_local(
|
||||
monkeypatch: pytest.MonkeyPatch, tmp_path: Path
|
||||
) -> None:
|
||||
"""Phase 38: DB rows of both kinds — the git row is cloned into
|
||||
``BOR_SOURCES_DIR``, the local row is its existing directory itself
|
||||
(no clone), in row order; the env list is ignored."""
|
||||
calls, fake = _fake_clone_factory()
|
||||
monkeypatch.setattr(import_docs, "clone_or_pull", fake)
|
||||
local_dir = tmp_path / "LocalDocs"
|
||||
local_dir.mkdir()
|
||||
(local_dir / "a.md").write_text("# A\nlocal fixture\n", encoding="utf-8")
|
||||
monkeypatch.setattr(
|
||||
import_docs,
|
||||
"effective_sources",
|
||||
lambda db: (
|
||||
[_git_row("https://db.example/only.git"), _local_row(str(local_dir))],
|
||||
"db",
|
||||
),
|
||||
)
|
||||
settings = _settings(
|
||||
git_sources="https://env.example/ignored.git",
|
||||
sources_dir=str(tmp_path / "bor"),
|
||||
)
|
||||
|
||||
sources = import_docs._resolve_sources(None, settings)
|
||||
|
||||
assert sources == [tmp_path / "bor" / "only", local_dir]
|
||||
assert calls == [("https://db.example/only.git", tmp_path / "bor" / "only")]
|
||||
|
||||
|
||||
def test_resolve_sources_missing_local_dir_aborts(
|
||||
monkeypatch: pytest.MonkeyPatch, tmp_path: Path
|
||||
) -> None:
|
||||
"""Phase 38: a local row whose directory is gone at run time →
|
||||
``GitSyncError`` naming the path, before any import (the same
|
||||
pre-import fail-loud as a failing git clone)."""
|
||||
monkeypatch.setattr(import_docs, "clone_or_pull", _fake_clone_factory()[1])
|
||||
missing = tmp_path / "Gone"
|
||||
monkeypatch.setattr(
|
||||
import_docs,
|
||||
"effective_sources",
|
||||
lambda db: ([_local_row(str(missing))], "db"),
|
||||
)
|
||||
settings = _settings(sources_dir=str(tmp_path / "bor"))
|
||||
|
||||
with pytest.raises(GitSyncError, match=f"local source missing: {re.escape(str(missing))}"):
|
||||
import_docs._resolve_sources(None, settings)
|
||||
|
||||
|
||||
def test_main_git_failure_aborts_before_import(
|
||||
monkeypatch: pytest.MonkeyPatch, tmp_path: Path, capsys: pytest.CaptureFixture[str]
|
||||
) -> None:
|
||||
@@ -236,7 +310,9 @@ def test_main_git_failure_aborts_before_import(
|
||||
)
|
||||
monkeypatch.setattr(import_docs, "get_settings", lambda: settings)
|
||||
monkeypatch.setattr(
|
||||
import_docs, "effective_git_sources", lambda db: (["https://host/a/bad.git"], "env")
|
||||
import_docs,
|
||||
"effective_sources",
|
||||
lambda db: ([_git_row("https://host/a/bad.git")], "env"),
|
||||
)
|
||||
|
||||
def failing_clone(url: str, dest: Path | str) -> Path:
|
||||
@@ -253,7 +329,34 @@ def test_main_git_failure_aborts_before_import(
|
||||
|
||||
assert rc == 1
|
||||
err = capsys.readouterr().err
|
||||
assert "import_docs: git sync failed" in err
|
||||
assert "import_docs: source sync failed" in err
|
||||
assert "bad.git" in err # the failing repo is named
|
||||
assert fake_import.calls == [] # no partial import
|
||||
assert not (tmp_path / "bor").exists()
|
||||
|
||||
|
||||
def test_main_missing_local_dir_aborts_before_import(
|
||||
monkeypatch: pytest.MonkeyPatch, tmp_path: Path, capsys: pytest.CaptureFixture[str]
|
||||
) -> None:
|
||||
"""Phase 38: a DB local row whose directory is missing → exit code
|
||||
1, ``local source missing: <path>`` on stderr, zero import attempts
|
||||
(no ``--source`` given, so the DB row is what should have been
|
||||
imported)."""
|
||||
settings = _settings(sources_dir=str(tmp_path / "bor"))
|
||||
monkeypatch.setattr(import_docs, "get_settings", lambda: settings)
|
||||
missing = tmp_path / "Gone"
|
||||
monkeypatch.setattr(
|
||||
import_docs,
|
||||
"effective_sources",
|
||||
lambda db: ([_local_row(str(missing))], "db"),
|
||||
)
|
||||
fake_import = FakeImportSources()
|
||||
monkeypatch.setattr(import_docs, "import_sources", fake_import)
|
||||
|
||||
rc = import_docs.main([])
|
||||
|
||||
assert rc == 1
|
||||
err = capsys.readouterr().err
|
||||
assert "import_docs: source sync failed" in err
|
||||
assert f"local source missing: {missing}" in err # the path is named
|
||||
assert fake_import.calls == [] # no partial import
|
||||
|
||||
@@ -0,0 +1,261 @@
|
||||
"""Integration: migration 0007 (git_sources.kind + path) schema contract.
|
||||
|
||||
Drives the **real Alembic engine** against the live dev database
|
||||
(``podman compose up -d db``), mirroring the style of
|
||||
``test_migration_0004.py`` / ``test_migration_0006.py``
|
||||
(information_schema assertions on the state the migration must leave).
|
||||
The tests target revision ``0007`` explicitly so later migrations
|
||||
cannot break them:
|
||||
|
||||
* upgrade 0006 → 0007 → ``git_sources`` gains ``kind TEXT NOT NULL``
|
||||
(server default ``'git'``, check constraint ``ck_git_sources_kind``:
|
||||
``kind IN ('git', 'local')``) and ``path TEXT`` (nullable) with the
|
||||
unique index ``uq_git_sources_path``; a row inserted before the
|
||||
upgrade (the pre-0007 insert shape) keeps ``kind='git'`` /
|
||||
``path=NULL`` after it;
|
||||
* the check constraint rejects any kind other than ``git``/``local``;
|
||||
* the unique index rejects duplicate local paths but tolerates NULL
|
||||
paths (git rows);
|
||||
* downgrade to 0006 → both columns, the constraint, and the index are
|
||||
gone;
|
||||
* upgrade back to 0007 → they are back (round-trip).
|
||||
|
||||
The ``alembic`` fixture guarantees the DB ends at head even if a test
|
||||
fails or the process is interrupted.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
from collections.abc import Iterator
|
||||
from typing import Any
|
||||
|
||||
import pytest
|
||||
from alembic.config import Config
|
||||
from sqlalchemy import text
|
||||
from sqlalchemy.exc import IntegrityError
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from alembic import command
|
||||
from app.db import db_available
|
||||
|
||||
URL_BASE = "https://git.example.com/mig0007"
|
||||
PATH_BASE = "/tmp/brain-of-reese-mig0007"
|
||||
|
||||
|
||||
@pytest.fixture()
|
||||
def alembic(db: Session) -> Iterator[Config]:
|
||||
"""Real Alembic config bound to the dev DB (URL from app settings).
|
||||
|
||||
Starts at head (repairs an interrupted earlier run); teardown upgrades
|
||||
to head no matter what happened, so the dev DB is never left below
|
||||
head.
|
||||
"""
|
||||
if not db_available():
|
||||
pytest.skip("Postgres not reachable — run `podman compose up -d db` first")
|
||||
cfg = Config() # no alembic.ini file — env.py gets the URL from app config
|
||||
cfg.set_main_option("script_location", "alembic")
|
||||
command.upgrade(cfg, "head")
|
||||
try:
|
||||
yield cfg
|
||||
finally:
|
||||
command.upgrade(cfg, "head")
|
||||
|
||||
|
||||
def _column(db: Session, column: str) -> tuple[Any, ...] | None:
|
||||
"""(data_type, is_nullable, column_default) for one git_sources column."""
|
||||
row = db.execute(
|
||||
text(
|
||||
"SELECT data_type, is_nullable, column_default"
|
||||
" FROM information_schema.columns"
|
||||
" WHERE table_name = 'git_sources' AND column_name = :c"
|
||||
),
|
||||
{"c": column},
|
||||
).fetchone()
|
||||
return tuple(row) if row is not None else None
|
||||
|
||||
|
||||
def _version(db: Session) -> str | None:
|
||||
return db.execute(text("SELECT version_num FROM alembic_version")).scalar()
|
||||
|
||||
|
||||
def _check_constraint_def(db: Session) -> str | None:
|
||||
"""Definition of ``ck_git_sources_kind``, or None if it does not exist.
|
||||
|
||||
Only call while ``git_sources`` exists (the ``::regclass`` cast errors
|
||||
otherwise).
|
||||
"""
|
||||
row = db.execute(
|
||||
text(
|
||||
"SELECT pg_get_constraintdef(oid) FROM pg_constraint"
|
||||
" WHERE conname = 'ck_git_sources_kind'"
|
||||
" AND conrelid = 'git_sources'::regclass"
|
||||
)
|
||||
).fetchone()
|
||||
return row[0] if row is not None else None
|
||||
|
||||
|
||||
def _unique_path_index(db: Session) -> int:
|
||||
"""1 iff ``uq_git_sources_path`` exists as a UNIQUE index."""
|
||||
count: Any = db.execute(
|
||||
text(
|
||||
"SELECT count(*) FROM pg_indexes"
|
||||
" WHERE tablename = 'git_sources' AND indexname = 'uq_git_sources_path'"
|
||||
" AND indexdef ILIKE 'CREATE UNIQUE%'"
|
||||
)
|
||||
).scalar()
|
||||
assert count is not None, "pg_indexes count must be an int"
|
||||
return int(count)
|
||||
|
||||
|
||||
def _insert(db: Session, url: str, kind: str | None = None, path: str | None = None) -> None:
|
||||
"""Insert one git_sources row; kind/path omitted → pre-0007 shape."""
|
||||
if kind is None and path is None:
|
||||
db.execute(
|
||||
text("INSERT INTO git_sources (id, url) VALUES (gen_random_uuid(), :u)"),
|
||||
{"u": url},
|
||||
)
|
||||
else:
|
||||
db.execute(
|
||||
text(
|
||||
"INSERT INTO git_sources (id, url, kind, path)"
|
||||
" VALUES (gen_random_uuid(), :u, :k, :p)"
|
||||
),
|
||||
{"u": url, "k": kind, "p": path},
|
||||
)
|
||||
db.commit()
|
||||
|
||||
|
||||
def _delete_by_url(db: Session, url: str) -> None:
|
||||
db.execute(text("DELETE FROM git_sources WHERE url = :u"), {"u": url})
|
||||
db.commit()
|
||||
|
||||
|
||||
def test_upgrade_to_0007_adds_kind_and_path(db: Session, alembic: Config) -> None:
|
||||
"""Upgrade 0006 → 0007: both columns exist with the locked types,
|
||||
nullability, and defaults, plus the CHECK constraint and the unique
|
||||
path index."""
|
||||
command.downgrade(alembic, "0006") # start from the pre-0007 state
|
||||
assert _version(db) == "0006"
|
||||
|
||||
command.upgrade(alembic, "0007")
|
||||
assert _version(db) == "0007", "alembic_version must be at 0007"
|
||||
|
||||
kind = _column(db, "kind")
|
||||
assert kind is not None, "git_sources.kind is missing"
|
||||
assert kind[0] == "text", "git_sources.kind must be TEXT"
|
||||
assert kind[1] == "NO", "git_sources.kind must be NOT NULL"
|
||||
assert kind[2] is not None and "'git'" in kind[2], (
|
||||
"git_sources.kind must have server default 'git'"
|
||||
)
|
||||
|
||||
path = _column(db, "path")
|
||||
assert path is not None, "git_sources.path is missing"
|
||||
assert path[0] == "text", "git_sources.path must be TEXT"
|
||||
assert path[1] == "YES", "git_sources.path must be NULLABLE"
|
||||
|
||||
constraint = _check_constraint_def(db)
|
||||
assert constraint is not None, "ck_git_sources_kind is missing"
|
||||
assert "git" in constraint and "local" in constraint, (
|
||||
f"ck_git_sources_kind must restrict kind to git|local, got: {constraint}"
|
||||
)
|
||||
|
||||
assert _unique_path_index(db) == 1, "uq_git_sources_path unique index is missing"
|
||||
|
||||
|
||||
def test_pre_0007_row_reads_as_git(db: Session, alembic: Config) -> None:
|
||||
"""A row inserted before the upgrade (url only — the pre-0007 insert
|
||||
shape) reads as ``kind='git'``, ``path=NULL`` after it."""
|
||||
command.downgrade(alembic, "0006")
|
||||
url = f"{URL_BASE}/pre-existing.git"
|
||||
_insert(db, url) # no kind/path columns exist at 0006
|
||||
try:
|
||||
command.upgrade(alembic, "0007")
|
||||
kind, path = db.execute(
|
||||
text("SELECT kind, path FROM git_sources WHERE url = :u"), {"u": url}
|
||||
).one()
|
||||
assert kind == "git", "a pre-0007 row must read as kind='git'"
|
||||
assert path is None, "a pre-0007 row must keep path=NULL"
|
||||
finally:
|
||||
_delete_by_url(db, url)
|
||||
|
||||
|
||||
def test_kind_defaults_to_git_for_new_inserts(db: Session, alembic: Config) -> None:
|
||||
"""An insert that omits kind (the API's pre-phase-38 shape) lands as
|
||||
``kind='git'`` via the server default."""
|
||||
command.upgrade(alembic, "head")
|
||||
url = f"{URL_BASE}/default-kind.git"
|
||||
_insert(db, url)
|
||||
try:
|
||||
kind, path = db.execute(
|
||||
text("SELECT kind, path FROM git_sources WHERE url = :u"), {"u": url}
|
||||
).one()
|
||||
assert kind == "git", "git_sources.kind must default to 'git'"
|
||||
assert path is None, "git_sources.path must default to NULL"
|
||||
finally:
|
||||
_delete_by_url(db, url)
|
||||
|
||||
|
||||
def test_check_constraint_rejects_unknown_kind(db: Session, alembic: Config) -> None:
|
||||
"""``ck_git_sources_kind`` is what later API validation relies on:
|
||||
any kind other than git|local raises IntegrityError."""
|
||||
command.upgrade(alembic, "head")
|
||||
with pytest.raises(IntegrityError):
|
||||
_insert(db, f"{URL_BASE}/bogus-kind.git", kind="bogus")
|
||||
db.rollback() # the IntegrityError aborts the open transaction
|
||||
|
||||
|
||||
def test_duplicate_local_path_rejected(db: Session, alembic: Config) -> None:
|
||||
"""The unique path index is what the API's 409 relies on: two local
|
||||
rows with the same path are rejected (NULL paths stay distinct —
|
||||
git rows are unaffected)."""
|
||||
command.upgrade(alembic, "head")
|
||||
url_a = f"{URL_BASE}/dup-path-a.git"
|
||||
url_b = f"{URL_BASE}/dup-path-b.git"
|
||||
url_c = f"{URL_BASE}/dup-path-c.git"
|
||||
path = f"{PATH_BASE}/shared"
|
||||
try:
|
||||
_insert(db, url_a, kind="local", path=path)
|
||||
with pytest.raises(IntegrityError):
|
||||
_insert(db, url_b, kind="local", path=path)
|
||||
db.rollback()
|
||||
# NULL paths are distinct under the unique index (git rows).
|
||||
_insert(db, url_b)
|
||||
_insert(db, url_c)
|
||||
finally:
|
||||
db.rollback()
|
||||
_delete_by_url(db, url_a)
|
||||
_delete_by_url(db, url_b)
|
||||
_delete_by_url(db, url_c)
|
||||
|
||||
|
||||
def test_downgrade_to_0006_drops_columns(db: Session, alembic: Config) -> None:
|
||||
"""Downgrade to 0006: both columns, the CHECK constraint, and the
|
||||
unique index are dropped (A13 — reversible)."""
|
||||
command.downgrade(alembic, "0006")
|
||||
assert _version(db) == "0006"
|
||||
|
||||
assert _column(db, "kind") is None, "git_sources.kind must be dropped"
|
||||
assert _column(db, "path") is None, "git_sources.path must be dropped"
|
||||
assert _check_constraint_def(db) is None, "ck_git_sources_kind must be dropped"
|
||||
assert _unique_path_index(db) == 0, "uq_git_sources_path must be dropped"
|
||||
|
||||
|
||||
def test_upgrade_round_trip_restores_columns(db: Session, alembic: Config) -> None:
|
||||
"""Downgrade to 0006, then upgrade back to 0007: columns, default,
|
||||
constraint, and index are back."""
|
||||
command.downgrade(alembic, "0006")
|
||||
command.upgrade(alembic, "0007")
|
||||
assert _version(db) == "0007", "round-trip upgrade must land at 0007"
|
||||
|
||||
kind = _column(db, "kind")
|
||||
assert kind is not None, "git_sources.kind must be back"
|
||||
assert kind[1] == "NO" and kind[2] is not None and "'git'" in kind[2], (
|
||||
"git_sources.kind must keep its NOT NULL 'git' default after the round-trip"
|
||||
)
|
||||
|
||||
path = _column(db, "path")
|
||||
assert path is not None and path[1] == "YES", "git_sources.path must be back"
|
||||
|
||||
constraint = _check_constraint_def(db)
|
||||
assert constraint is not None, "ck_git_sources_kind must be back"
|
||||
|
||||
assert _unique_path_index(db) == 1, "uq_git_sources_path must be back"
|
||||
@@ -1,20 +1,31 @@
|
||||
"""Integration: the admin sources-sync API (phase 32, task 01; phase 35,
|
||||
task 03 re-points the URL resolution at the shared resolver).
|
||||
task 03 re-points the URL resolution at the shared resolver; phase 38,
|
||||
task 03 adds the local kind).
|
||||
|
||||
Covers the in-process sync runner end to end over HTTP: anonymous 403s
|
||||
on both endpoints; admin idle → 202 → ``success`` with the full
|
||||
ImportSummary detail; 409 on a double trigger while a run is in flight;
|
||||
``GitSyncError`` → ``failed`` with the failing repo named and **zero**
|
||||
import attempts; empty on *both* origins (``git_sources`` table +
|
||||
``BOR_GIT_SOURCES``) → ``failed`` loudly; an embedding failure →
|
||||
``failed`` with any credentials masked; the import always runs with
|
||||
``prune=True``; and the phase-31 overview trigger is change-gated (no
|
||||
``lite`` call on an unchanged KB).
|
||||
import attempts; empty on *both* origins (no git rows, no local rows,
|
||||
no env URLs) → ``failed`` loudly (``no sources configured
|
||||
(git or local)``); an embedding failure → ``failed`` with any
|
||||
credentials masked; the import always runs with ``prune=True``; and the
|
||||
phase-31 overview trigger is change-gated (no ``lite`` call on an
|
||||
unchanged KB).
|
||||
|
||||
Phase 35: the runner resolves the repos through
|
||||
:func:`app.rag.git_sources.effective_git_sources` — the **real**
|
||||
resolver against the **real** ``git_sources`` table (truncated around
|
||||
every test), so DB-over-env and the env fallback go through the actual
|
||||
Phase 38 (local kind): local-only, git-only, and mixed syncs over a
|
||||
**host temp local dir** (the app server runs on the same host) — the
|
||||
mixed run goes through the **real** ``import_sources`` (deterministic
|
||||
in-process ``FakeEmbedder``, no network), so the local file verifiably
|
||||
lands in the KB via ``GET /api/docs`` and union pruning holds (a file
|
||||
deleted out of the local dir is pruned on the next sync while the git
|
||||
doc survives); a local directory missing at sync time → ``failed`` with
|
||||
``local source missing: <path>`` and **zero** import attempts.
|
||||
|
||||
Phase 35: the runner resolves the sources through
|
||||
:func:`app.rag.git_sources.effective_sources` — the **real** resolver
|
||||
against the **real** ``git_sources`` table (truncated around every
|
||||
test), so DB-over-env and the env fallback go through the actual
|
||||
indirection; the env list is driven by a fresh ``Settings`` on the
|
||||
resolver's module (the dev ``.env`` never leaks in).
|
||||
|
||||
@@ -34,7 +45,7 @@ import asyncio
|
||||
import logging
|
||||
import time
|
||||
from collections.abc import Iterator
|
||||
from datetime import datetime
|
||||
from datetime import UTC, datetime
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
@@ -51,6 +62,7 @@ from app.rag.importer import ImportSummary
|
||||
from app.rag.llm import EmbeddingError, LLMClient
|
||||
from scripts.git_sync import GitSyncError
|
||||
from tests.conftest import ADMIN_PASSWORD
|
||||
from tests.fakes import FakeEmbedder
|
||||
|
||||
|
||||
@pytest.fixture(autouse=True)
|
||||
@@ -110,6 +122,30 @@ def _seed(db: Session, url: str) -> None:
|
||||
db.commit()
|
||||
|
||||
|
||||
def _seed_local(db: Session, path: Path) -> None:
|
||||
"""A ``kind=local`` row as the phase-38 API stores it: the expanded
|
||||
absolute path in both ``path`` and the NOT-NULL ``url`` column."""
|
||||
db.add(GitSource(url=str(path), kind="local", path=str(path)))
|
||||
db.commit()
|
||||
|
||||
|
||||
@pytest.fixture()
|
||||
def clean_documents(db: Session) -> Iterator[None]:
|
||||
"""The real-import tests write ``documents``/``chunks`` (the canonical
|
||||
KB state) — global, truncated around every such test."""
|
||||
db.execute(text("TRUNCATE chunks, documents"))
|
||||
db.commit()
|
||||
yield
|
||||
db.execute(text("TRUNCATE chunks, documents"))
|
||||
db.commit()
|
||||
|
||||
|
||||
def _real_llm(monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
"""The pipeline's ``LLMClient`` becomes the deterministic in-process
|
||||
``FakeEmbedder`` (real import, no network)."""
|
||||
monkeypatch.setattr(sync_api, "LLMClient", lambda: FakeEmbedder())
|
||||
|
||||
|
||||
def _login(client: TestClient) -> None:
|
||||
r = client.post("/api/login", json={"password": ADMIN_PASSWORD})
|
||||
assert r.status_code == 204, f"admin login failed: {r.status_code} {r.text}"
|
||||
@@ -372,11 +408,12 @@ def test_git_failure_marks_failed_and_skips_import(
|
||||
_poll(sync_client, "failed")
|
||||
|
||||
|
||||
def test_no_git_sources_configured_fails_loudly(
|
||||
def test_no_sources_configured_fails_loudly(
|
||||
sync_client: TestClient, monkeypatch: pytest.MonkeyPatch, tmp_path: Path
|
||||
) -> None:
|
||||
"""Both origins empty (the truncate fixture + a blank env) → the
|
||||
fail-loud error names *both* (phase 35)."""
|
||||
"""Both origins empty — no git rows, no local rows, no env URLs
|
||||
(the truncate fixture + a blank env) → the fail-loud error
|
||||
(phase 38: git-only message retired)."""
|
||||
# Whitespace-only is just as unconfigured as empty.
|
||||
_stub_env(monkeypatch, " , ")
|
||||
monkeypatch.setattr(
|
||||
@@ -393,9 +430,7 @@ def test_no_git_sources_configured_fails_loudly(
|
||||
assert sync_client.post("/api/sync").status_code == 202
|
||||
|
||||
body = _poll(sync_client, "failed")
|
||||
assert body["error"] == (
|
||||
"no git sources configured (git_sources table empty and BOR_GIT_SOURCES unset)"
|
||||
)
|
||||
assert body["error"] == "no sources configured (git or local)"
|
||||
assert clone_calls == [] # git is never touched
|
||||
assert fake_import.sources == []
|
||||
|
||||
@@ -464,6 +499,141 @@ def test_env_fallback_when_table_empty(
|
||||
assert any("sync: started repos=1 origin=env" in r.getMessage() for r in caplog.records)
|
||||
|
||||
|
||||
# --- phase 38: the local kind (real import, host temp dirs) ---------------
|
||||
|
||||
|
||||
def test_local_only_sync_imports_dir(
|
||||
sync_client: TestClient,
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
db: Session,
|
||||
tmp_path: Path,
|
||||
clean_documents: None,
|
||||
) -> None:
|
||||
"""Local-only config: the host temp dir (one fixture ``.md``) is
|
||||
walked directly — no clone at all — and the file lands in the KB
|
||||
(``GET /api/docs`` as admin)."""
|
||||
local_dir = tmp_path / "LocalDocs"
|
||||
local_dir.mkdir()
|
||||
(local_dir / "notes.md").write_text("# Local Notes\nthe local fixture\n", encoding="utf-8")
|
||||
_seed_local(db, local_dir)
|
||||
_stub_env(monkeypatch) # env must not matter once the table has a row
|
||||
monkeypatch.setattr(
|
||||
sync_api,
|
||||
"get_settings",
|
||||
lambda: _settings(sources_dir=str(tmp_path / "bor")),
|
||||
)
|
||||
clone_calls, fake_clone = _fake_clone()
|
||||
monkeypatch.setattr(sync_api, "clone_or_pull", fake_clone)
|
||||
_real_llm(monkeypatch) # real import_sources, deterministic embeddings
|
||||
|
||||
_login(sync_client)
|
||||
assert sync_client.post("/api/sync").status_code == 202
|
||||
body = _poll(sync_client, "success")
|
||||
|
||||
assert clone_calls == [] # nothing to clone — local is walked directly
|
||||
assert body["detail"]["added"] == 1
|
||||
assert body["detail"]["errors"] == 0
|
||||
# The local file is in the KB, sourced by the directory's basename.
|
||||
docs = sync_client.get("/api/docs").json()["documents"]
|
||||
assert [(d["source"], d["path"]) for d in docs] == [("LocalDocs", "notes.md")]
|
||||
|
||||
|
||||
def test_mixed_git_local_sync_imports_both_and_prunes_union(
|
||||
sync_client: TestClient,
|
||||
monkeypatch: pytest.MonkeyPatch,
|
||||
db: Session,
|
||||
tmp_path: Path,
|
||||
caplog: pytest.LogCaptureFixture,
|
||||
clean_documents: None,
|
||||
) -> None:
|
||||
"""Mixed config: the git row is cloned, the local dir walked, and
|
||||
both are imported in one run over the single combined list. The
|
||||
started log carries the kind counts; a file deleted out of the
|
||||
local dir is pruned on the next sync (union prune) while the git
|
||||
doc survives."""
|
||||
repo_url = f"file://{tmp_path / 'repo.git'}"
|
||||
_seed(db, repo_url)
|
||||
local_dir = tmp_path / "LocalDocs"
|
||||
local_dir.mkdir()
|
||||
(local_dir / "a.md").write_text("# A\nfirst local file\n", encoding="utf-8")
|
||||
(local_dir / "b.md").write_text("# B\nsecond local file\n", encoding="utf-8")
|
||||
_seed_local(db, local_dir)
|
||||
_stub_env(monkeypatch)
|
||||
monkeypatch.setattr(
|
||||
sync_api,
|
||||
"get_settings",
|
||||
lambda: _settings(sources_dir=str(tmp_path / "bor")),
|
||||
)
|
||||
clone_calls, fake_clone = _fake_clone()
|
||||
monkeypatch.setattr(sync_api, "clone_or_pull", fake_clone)
|
||||
_real_llm(monkeypatch)
|
||||
|
||||
_login(sync_client)
|
||||
with caplog.at_level(logging.INFO, logger="app.api.sync"):
|
||||
assert sync_client.post("/api/sync").status_code == 202
|
||||
body = _poll(sync_client, "success")
|
||||
|
||||
# Git cloned into BOR_SOURCES_DIR, local dir walked in row order.
|
||||
assert clone_calls == [(repo_url, tmp_path / "bor" / "repo")]
|
||||
assert body["detail"]["added"] == 3
|
||||
assert any(
|
||||
"sync: started repos=2 origin=db git=1 local=1" in r.getMessage() for r in caplog.records
|
||||
)
|
||||
docs = {(d["source"], d["path"]) for d in sync_client.get("/api/docs").json()["documents"]}
|
||||
assert docs == {("repo", "notes.md"), ("LocalDocs", "a.md"), ("LocalDocs", "b.md")}
|
||||
|
||||
# Union prune: delete one local file → the next sync prunes exactly
|
||||
# it; the git doc (and the surviving local file) stay.
|
||||
(local_dir / "b.md").unlink()
|
||||
assert sync_client.post("/api/sync").status_code == 202
|
||||
body = _poll(sync_client, "success")
|
||||
assert body["detail"]["pruned"] == 1
|
||||
assert body["detail"]["added"] == 0
|
||||
docs = {(d["source"], d["path"]) for d in sync_client.get("/api/docs").json()["documents"]}
|
||||
assert docs == {("repo", "notes.md"), ("LocalDocs", "a.md")}
|
||||
|
||||
|
||||
def test_missing_local_dir_fails_loudly_and_imports_nothing(
|
||||
sync_client: TestClient, monkeypatch: pytest.MonkeyPatch, db: Session, tmp_path: Path
|
||||
) -> None:
|
||||
"""A local row whose directory is gone at sync time (moved/deleted
|
||||
since add-time) → ``failed`` naming the path, **zero** import
|
||||
attempts — the git row before it in row order was still cloned
|
||||
(per-row walk; a clone is not an import)."""
|
||||
repo_url = f"file://{tmp_path / 'repo.git'}"
|
||||
missing = tmp_path / "Gone"
|
||||
db.add(
|
||||
GitSource(url=repo_url, kind="git", added_at=datetime(2026, 1, 1, tzinfo=UTC))
|
||||
)
|
||||
db.add(
|
||||
GitSource(url=str(missing), kind="local", path=str(missing),
|
||||
added_at=datetime(2026, 1, 2, tzinfo=UTC))
|
||||
)
|
||||
db.commit()
|
||||
_stub_env(monkeypatch)
|
||||
monkeypatch.setattr(
|
||||
sync_api,
|
||||
"get_settings",
|
||||
lambda: _settings(sources_dir=str(tmp_path / "bor")),
|
||||
)
|
||||
clone_calls, fake_clone = _fake_clone()
|
||||
monkeypatch.setattr(sync_api, "clone_or_pull", fake_clone)
|
||||
fake_import = FakeImportSources(ImportSummary())
|
||||
monkeypatch.setattr(sync_api, "import_sources", fake_import)
|
||||
fake_overview = FakeOverview(ok=True)
|
||||
monkeypatch.setattr(sync_api, "regenerate_overview", fake_overview)
|
||||
|
||||
_login(sync_client)
|
||||
assert sync_client.post("/api/sync").status_code == 202
|
||||
|
||||
body = _poll(sync_client, "failed")
|
||||
assert f"local source missing: {missing}" in body["error"] # the path is named
|
||||
assert body["detail"] == {}
|
||||
assert clone_calls == [(repo_url, tmp_path / "bor" / "repo")] # git row walked first
|
||||
assert fake_import.sources == [] # no partial import
|
||||
assert fake_overview.llms == []
|
||||
|
||||
|
||||
def test_import_error_is_reported_with_credentials_masked(
|
||||
sync_client: TestClient, monkeypatch: pytest.MonkeyPatch, db: Session, tmp_path: Path
|
||||
) -> None:
|
||||
|
||||
@@ -1,12 +1,16 @@
|
||||
"""Unit: the shared git-source resolver (phase 35, task 03).
|
||||
"""Unit: the shared source resolver (phase 35, task 03; local kind,
|
||||
phase 38, task 03).
|
||||
|
||||
``effective_git_sources`` is driven with a stubbed session (no Postgres)
|
||||
``effective_sources`` is driven with a stubbed session (no Postgres)
|
||||
and a fresh ``Settings(_env_file=None)`` env (monkeypatched into the
|
||||
resolver's module — the dev ``.env`` never leaks in, same pattern as the
|
||||
integration suites): DB rows win in ``(added_at, id)`` order (env
|
||||
ignored), the ``BOR_GIT_SOURCES`` list is a fallback only while the
|
||||
table is empty (phase-28 CSV parse reused), and both-empty yields
|
||||
``([], "env")`` so the callers keep their fail-loud behavior.
|
||||
resolver's module — the dev ``.env`` never leaks in, same pattern as
|
||||
the integration suites): DB rows of **both kinds** win in
|
||||
``(added_at, id)`` order (env ignored), the ``BOR_GIT_SOURCES`` list is
|
||||
a git-only fallback while the table is empty (surfaced as synthetic
|
||||
``kind='git'`` rows, phase-28 CSV parse reused), and both-empty yields
|
||||
``([], "env")`` so the callers keep their fail-loud behavior. The
|
||||
phase-35 ``effective_git_sources`` alias is kept covered as well: it
|
||||
returns the repo URLs of the git rows only, same origin.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
@@ -52,33 +56,70 @@ def _stub_env(monkeypatch: pytest.MonkeyPatch, git_sources: str) -> None:
|
||||
)
|
||||
|
||||
|
||||
def _git(url: str) -> GitSource:
|
||||
return GitSource(url=url, kind="git")
|
||||
|
||||
|
||||
def _local(path: str) -> GitSource:
|
||||
"""A local row as phase-38 task 02 stores it: the expanded path in
|
||||
both ``path`` and the NOT-NULL ``url`` location column."""
|
||||
return GitSource(url=path, kind="local", path=path)
|
||||
|
||||
|
||||
# --- the three branches ----------------------------------------------------
|
||||
|
||||
|
||||
def test_db_rows_win_over_env(monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
"""Seeded table + env set → the DB list, origin ``db`` (env ignored)."""
|
||||
"""Seeded table + env set → the DB rows, origin ``db`` (env ignored)."""
|
||||
_stub_env(monkeypatch, "https://env.example/ignored.git")
|
||||
session = _FakeSession([GitSource(url="https://db.example/a.git"), GitSource(url="https://db.example/b.git")])
|
||||
session = _FakeSession(
|
||||
[GitSource(url="https://db.example/a.git"), GitSource(url="https://db.example/b.git")]
|
||||
)
|
||||
|
||||
urls, origin = resolver.effective_git_sources(session) # pyright: ignore[reportArgumentType]
|
||||
rows, origin = resolver.effective_sources(session) # pyright: ignore[reportArgumentType]
|
||||
|
||||
assert urls == ["https://db.example/a.git", "https://db.example/b.git"]
|
||||
assert [row.url for row in rows] == ["https://db.example/a.git", "https://db.example/b.git"]
|
||||
assert origin == "db"
|
||||
|
||||
|
||||
def test_env_fallback_while_table_empty(monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
"""Empty table + env set → the env list, origin ``env``.
|
||||
def test_mixed_kinds_returned_in_row_order(monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
"""Phase 38: the table holds git **and** local rows — both come back,
|
||||
in ``(added_at, id)`` row order, kinds and paths intact, env ignored."""
|
||||
_stub_env(monkeypatch, "https://env.example/ignored.git")
|
||||
local = "/abs/notes"
|
||||
session = _FakeSession(
|
||||
[_git("https://db.example/a.git"), _local(local), _git("git@db.example:b.git")]
|
||||
)
|
||||
|
||||
The CSV parse is ``Settings.git_source_list`` itself (phase 28):
|
||||
rows, origin = resolver.effective_sources(session) # pyright: ignore[reportArgumentType]
|
||||
|
||||
assert origin == "db"
|
||||
assert [row.kind for row in rows] == ["git", "local", "git"]
|
||||
assert [row.url for row in rows] == [
|
||||
"https://db.example/a.git",
|
||||
local,
|
||||
"git@db.example:b.git",
|
||||
]
|
||||
assert [row.path for row in rows] == [None, local, None]
|
||||
|
||||
|
||||
def test_env_fallback_git_only_while_table_empty(monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
"""Empty table + env set → synthetic ``kind='git'`` rows, origin
|
||||
``env``.
|
||||
|
||||
The env fallback is git-only (no local rows can come from it); the
|
||||
CSV parse is ``Settings.git_source_list`` itself (phase 28):
|
||||
whitespace-trimmed, empty entries dropped, order preserved.
|
||||
"""
|
||||
_stub_env(monkeypatch, " https://env.example/one.git , ,git@env.example:two.git ")
|
||||
session = _FakeSession([])
|
||||
|
||||
urls, origin = resolver.effective_git_sources(session) # pyright: ignore[reportArgumentType]
|
||||
rows, origin = resolver.effective_sources(session) # pyright: ignore[reportArgumentType]
|
||||
|
||||
assert urls == ["https://env.example/one.git", "git@env.example:two.git"]
|
||||
assert origin == "env"
|
||||
assert [row.url for row in rows] == ["https://env.example/one.git", "git@env.example:two.git"]
|
||||
assert all(row.kind == "git" for row in rows)
|
||||
assert all(row.path is None for row in rows)
|
||||
|
||||
|
||||
def test_both_empty_returns_empty_env(monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
@@ -86,9 +127,9 @@ def test_both_empty_returns_empty_env(monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
_stub_env(monkeypatch, " , ") # whitespace-only is just as unconfigured as empty
|
||||
session = _FakeSession([])
|
||||
|
||||
urls, origin = resolver.effective_git_sources(session) # pyright: ignore[reportArgumentType]
|
||||
rows, origin = resolver.effective_sources(session) # pyright: ignore[reportArgumentType]
|
||||
|
||||
assert urls == []
|
||||
assert rows == []
|
||||
assert origin == "env"
|
||||
|
||||
|
||||
@@ -99,12 +140,42 @@ def test_db_rows_ordered_by_added_at_then_id(monkeypatch: pytest.MonkeyPatch) ->
|
||||
"""The statement orders by ``(added_at, id)`` — oldest first, the id
|
||||
tie-break deciding same-timestamp inserts (matches the API's GET)."""
|
||||
_stub_env(monkeypatch, "")
|
||||
session = _FakeSession([GitSource(url="https://db.example/a.git")])
|
||||
session = _FakeSession([_git("https://db.example/a.git")])
|
||||
|
||||
resolver.effective_git_sources(session) # pyright: ignore[reportArgumentType]
|
||||
resolver.effective_sources(session) # pyright: ignore[reportArgumentType]
|
||||
|
||||
assert len(session.statements) == 1
|
||||
compiled = str(session.statements[0].compile(compile_kwargs={"literal_binds": True}))
|
||||
assert re.search(
|
||||
r"ORDER BY\s+git_sources\.added_at ASC,\s+git_sources\.id ASC", compiled
|
||||
), compiled
|
||||
|
||||
|
||||
# --- the phase-35 back-compat alias ----------------------------------------
|
||||
|
||||
|
||||
def test_alias_returns_git_urls_only(monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
"""``effective_git_sources`` (phase-35 name) — repo URLs of the git
|
||||
rows only; local rows are filtered out (they carry no clone URL),
|
||||
origin unchanged."""
|
||||
_stub_env(monkeypatch, "")
|
||||
session = _FakeSession(
|
||||
[_git("https://db.example/a.git"), _local("/abs/notes"), _git("git@db.example:b.git")]
|
||||
)
|
||||
|
||||
urls, origin = resolver.effective_git_sources(session) # pyright: ignore[reportArgumentType]
|
||||
|
||||
assert urls == ["https://db.example/a.git", "git@db.example:b.git"]
|
||||
assert origin == "db"
|
||||
|
||||
|
||||
def test_alias_env_fallback_unchanged(monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
"""The alias's env fallback is byte-identical to the phase-35 one
|
||||
(git-only CSV list, origin ``env``)."""
|
||||
_stub_env(monkeypatch, " https://env.example/one.git , ")
|
||||
session = _FakeSession([])
|
||||
|
||||
urls, origin = resolver.effective_git_sources(session) # pyright: ignore[reportArgumentType]
|
||||
|
||||
assert urls == ["https://env.example/one.git"]
|
||||
assert origin == "env"
|
||||
|
||||
Reference in New Issue
Block a user