feat(admin): local directory sources — kind/path on git_sources, combined sync + import, page form + badges
An existing, non-git directory is now a first-class source alongside
the git repos: one table (git_sources + kind discriminator — A13
reversible migration), one admin page, one Sync button (phase locked
decisions; the phase-35 table is extended, not duplicated). The DB is
the local-source registry — no env var for local paths;
BOR_GIT_SOURCES stays a git-only empty-table fallback.
Migration 0007 (reversible, up/down integration-tested):
git_sources.kind TEXT NOT NULL DEFAULT 'git' + ck_git_sources_kind
(kind IN ('git','local')); git_sources.path TEXT NULL +
uq_git_sources_path (mirrors 0006's uq_git_sources_url). Existing rows
read kind='git', path=NULL.
API (phase-35 contract extended, git byte-identical): POST kind=local
requires path — trimmed, ~-expanded, absolute + an existing server
directory, else 422 naming the path (fail loud at add-time); duplicate
path 409 (named); wrong field combos 422. GET rows carry kind + path
(git and env rows: path null); anonymous still 403 on every route (A10).
Sync + import_docs resolve DB git + local rows together: git →
clone_or_pull (unchanged); local → re-verified .is_dir() AT SYNC TIME
(it may have moved/deleted since add-time) — a missing dir raises
"local source missing: <path>" (sanitized) before anything imports;
one import_sources(..., prune=True) over the single combined list
(pruning covers the union). Both-empty fails loudly ("no sources
configured (git or local)"); --source still wins; the env fallback
stays git-only.
Page: second "Add a local directory" form (the same §7.4 never-stale
button + inline-error lifecycle as the git form; 422/409 details name
the path), Git/Local badges on rows (text + color, never color alone —
WCAG), updated hint (git + local together, union prune); the
anonymous sign-in gate is unchanged.
Tests: 0007 up/down; the API local-kind matrix (403/201/422/409) with
the git-kind suite green unchanged; the sync pipeline local/git/
mixed/missing against a host temp dir (the KB actually updated);
import_docs DB resolution + --source precedence. Story E2E (isolated,
deterministic across runs): add (Local badge) → missing path inline
422 naming it / duplicate 409 → the real Sync button imports the
fixture file (GET /api/docs + sentinel in its content) → file deleted
+ sync prunes it (union prune) → row removed; anonymous gate + 403s
(phase-35 regression). test_git_sources_admin.py (phase 35) green
UNCHANGED — no selector collision with the new form;
test_sync_button.py green.
Docs: README — the two managed kinds (git = clone/pull mirror; local =
direct in-place walk), add-time validation, union pruning, "the DB is
the local-source registry (no env var for local paths)";
.env.example — the env fallback is git-only.
This commit is contained in:
@@ -63,23 +63,29 @@ uv run python -m scripts.llm_probe # sanity: models + 768-dim check
|
||||
uv run python -m scripts.import_docs # import the configured sources (below)
|
||||
```
|
||||
|
||||
Two source modes:
|
||||
Two managed source kinds (one page, one registry) plus a manual override:
|
||||
|
||||
- **Git sources (recommended)** — the list is managed on the **admin
|
||||
Git sources page** (`/git-sources.html`) and stored in Postgres (see
|
||||
- **Git sources** — managed on the **admin Git sources page**
|
||||
(`/git-sources.html`) and stored in Postgres (see
|
||||
[Git-based sources](#git-based-sources)). `import_docs` clones each repo
|
||||
(first run) or pulls it (subsequent runs) into
|
||||
`BOR_SOURCES_DIR/<repo-name>/` (default `~/bor-sources`) and indexes the
|
||||
checkouts. While the stored list is empty, the `BOR_GIT_SOURCES` variable
|
||||
in `.env` is the fallback — the moment the page stores a source, the
|
||||
variable is ignored.
|
||||
- **Local directory sources** (phase 38) — an existing, non-git directory
|
||||
on the server, registered on the *same* admin page (see
|
||||
[Local directory sources](#local-directory-sources)). No clone, no
|
||||
checkout copy: the directory is walked in place. There is **no env var
|
||||
for local paths** — the DB is the registry.
|
||||
- **Manual directories** — `--source <path>` (repeatable) imports local
|
||||
directories directly and *always wins* over the git sources (stored list
|
||||
or env).
|
||||
directories directly and *always wins* over the stored sources (git and
|
||||
local) and the env fallback.
|
||||
- If neither is set (stored list, `--source`, and `BOR_GIT_SOURCES` all
|
||||
empty), the import falls back to the **previous** default,
|
||||
empty), `import_docs` falls back to the **previous** default,
|
||||
`~/Homelab` + `~/Deployments` — kept only for backwards compatibility,
|
||||
now replaced by the git sources list.
|
||||
now replaced by the managed sources; the UI Sync button instead fails
|
||||
loudly ("no sources configured (git or local)").
|
||||
|
||||
### 6. Run the app
|
||||
```bash
|
||||
@@ -107,12 +113,18 @@ uv run uvicorn app.main:app --reload
|
||||
new tab). **Admin-only** — anonymous visitors see a sign-in gate instead
|
||||
(the catalog is what the login locks; the document viewer itself stays
|
||||
open to everyone).
|
||||
- **Git sources** (`/git-sources.html`) — the list of git repositories the
|
||||
**Sync sources** button clones and indexes; **admin-only** (the same
|
||||
sign-in gate as Sources). Add or remove repositories here — no `.env`
|
||||
editing, no restart. Adding/removing does not clone or prune on its
|
||||
own: the Sync button performs that, and a removed repository's documents
|
||||
leave the index on the next sync.
|
||||
- **Git sources** (`/git-sources.html`) — the admin-managed source
|
||||
registry: the git repositories the **Sync sources** button clones and
|
||||
indexes, **and** existing local directories it imports directly
|
||||
(phase 38 — one table with a `kind` discriminator, one page); **admin-only**
|
||||
(the same sign-in gate as Sources). Add or remove sources here — no
|
||||
`.env` editing, no restart. A local directory must be an absolute,
|
||||
existing directory at add-time (a missing/relative path is rejected
|
||||
inline, naming the path; so are duplicates); list rows carry a **Git**
|
||||
or **Local** badge. Adding/removing does not clone or prune on its
|
||||
own: the Sync button performs that (git + local together, one run,
|
||||
prune over the union), and a removed source's documents leave the index
|
||||
on the next sync.
|
||||
|
||||
## Thinking
|
||||
|
||||
@@ -345,16 +357,52 @@ BOR_SOURCES_DIR=~/bor-sources # default; each repo lands in <dir>/<repo-name>/
|
||||
**nothing** (no partial junk). Fix the URL/connectivity and re-run — the
|
||||
other checkouts stay on disk and are pulled as usual.
|
||||
|
||||
### Local directory sources
|
||||
|
||||
Not every set of notes lives in a git repo — a plain directory can be a
|
||||
first-class source too (phase 38). It shares the git sources' **one
|
||||
table** (the `git_sources` registry with a `kind` discriminator: `git` |
|
||||
`local`, migration 0007), **one admin page**, and **one Sync button**:
|
||||
|
||||
- **Add it on the Git sources page** — the “Add a local directory” form
|
||||
next to the git form. Add-time validation fails loud: the path is
|
||||
trimmed, `~` is expanded, and must be an **absolute, existing directory
|
||||
on the server** — anything else (missing, relative, a file) is rejected
|
||||
with the path named inline; a duplicate path is rejected the same way.
|
||||
There is **no env var for local paths** — the DB is the local-source
|
||||
registry (`BOR_GIT_SOURCES` stays a git-only fallback).
|
||||
- **Sync walks it directly** — no clone, no checkout copy: each run
|
||||
indexes the directory in place (A9 format filter, hidden-dir skip,
|
||||
sha256 delta), together with the git checkouts in the **same run**.
|
||||
`documents.source` is the directory's name. The directory is
|
||||
re-verified to exist **at sync time** (it may have moved or been
|
||||
deleted since add-time): a missing directory fails the run loudly,
|
||||
naming the path, and imports **nothing** (the same pre-import fail-loud
|
||||
as a failing git clone).
|
||||
- **Pruning is over the union** — git checkouts and local directories are
|
||||
imported together with `prune=True`, so a file removed from a local
|
||||
directory, a repo, or a removed source leaves the index on that run.
|
||||
Removing the row on the page stops the directory being a source; its
|
||||
documents leave the index on the next sync (exactly like git sources).
|
||||
- **`import_docs`** (no `--source`) resolves the stored git **and** local
|
||||
rows — git cloned/pulled as above, local walked directly — in one run;
|
||||
`--source` still wins over everything; while the table is empty,
|
||||
`BOR_GIT_SOURCES` is the git-only fallback; no git rows, no local rows,
|
||||
and no env URLs fails loudly ("no sources configured (git or local)").
|
||||
|
||||
### Sync from the UI
|
||||
|
||||
The **Sync sources** button on the **Sources** page — visible to the
|
||||
**admin only** (anonymous visitors never see it) — runs the whole
|
||||
git-source refresh in one click, in-process:
|
||||
|
||||
1. **clone/pull** every configured git source — the admin-managed list
|
||||
(the `git_sources` table; `BOR_GIT_SOURCES` only while that list is
|
||||
empty), through the same `clone_or_pull` the CLI uses (shallow clone
|
||||
on first run, `git pull --ff-only` afterwards);
|
||||
1. **clone/pull + walk** every configured source — the git sources (the
|
||||
admin-managed `git_sources` table; `BOR_GIT_SOURCES` only while that
|
||||
list is empty) through the same `clone_or_pull` the CLI uses (shallow
|
||||
clone on first run, `git pull --ff-only` afterwards), **and** the
|
||||
local directories registered on the same page, walked directly
|
||||
(re-verified to exist at sync time — a missing directory fails the run
|
||||
loudly, naming the path);
|
||||
2. **re-import with prune** — the `--prune` equivalent, so files deleted
|
||||
upstream leave the index (the button is the canonical "mirror the
|
||||
repos" action); the sha256 delta still skips unchanged files, so an
|
||||
@@ -363,12 +411,12 @@ git-source refresh in one click, in-process:
|
||||
chat turn injects) — but only when the import actually changed the
|
||||
knowledge base.
|
||||
|
||||
- **Prerequisites:** at least one git source must be configured — a row
|
||||
on the admin Git sources page, or `BOR_GIT_SOURCES` in `.env` while the
|
||||
stored list is empty; **both** empty fails the sync loudly ("no git
|
||||
sources configured"), because the button targets the git repos only
|
||||
(manual `--source` directories have no repo to clone) — and `git` must
|
||||
be on the app's `PATH`.
|
||||
- **Prerequisites:** at least one source must be configured — a git or
|
||||
local row on the admin Git sources page, or `BOR_GIT_SOURCES` in `.env`
|
||||
while the stored list is empty (git-only); **all** empty fails the sync
|
||||
loudly ("no sources configured (git or local)"), because the button
|
||||
targets the admin-managed registry (manual `--source` directories have
|
||||
no place in it) — and `git` must be on the app's `PATH` for git sources.
|
||||
- **States:** clicking starts the run (`202`) and the button goes
|
||||
disabled with **Syncing…** (spinning icon) while the page polls
|
||||
`GET /api/sync/status` every 2 s. There is deliberately **no
|
||||
@@ -597,7 +645,7 @@ served locally (no CDN), `BOR_ENVIRONMENT=production`.
|
||||
| `BOR_AGENT_LIST_CALLS` | `1` | per-turn `list_documents` tool opportunities on grounded turns (0 disables the tool) |
|
||||
| `BOR_AGENT_READ_CALLS` | `1` | per-turn `read_document` tool opportunities on grounded turns (0 disables the tool) |
|
||||
| `BOR_IMPORT_EXTENSIONS` | `md,markdown,txt,yaml,yml,json,py` | csv of importable formats (may only narrow the A9 set) |
|
||||
| `BOR_GIT_SOURCES` | — (empty) | csv of git repo URLs — **fallback while the admin Git sources page's list (Postgres `git_sources`) is empty**; the page is the primary management surface (see *Git-based sources*) |
|
||||
| `BOR_GIT_SOURCES` | — (empty) | csv of git repo URLs — **fallback while the admin Git sources page's list (Postgres `git_sources`) is empty**; the page is the primary management surface (see *Git-based sources*). **Git-only**: local directory sources have no env var — they are registered on the admin page (see *Local directory sources*) |
|
||||
| `BOR_SOURCES_DIR` | `~/bor-sources` | where the git source repos are cloned/pulled (one subdirectory per repo) |
|
||||
| `BOR_STEERING_MAX_CHARS` | `8000` | char budget for the `<tuning>` (steering notes) prompt section |
|
||||
| `BOR_SUMMARY_MAX_CHARS` | `12000` | cap on document content sent to the `lite` summary model at import (see *Document summaries*) |
|
||||
|
||||
Reference in New Issue
Block a user