Experimental/brain-of-reese
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat(sources): admin page to add and remove git sources (TODO.md L4)

Browse Source
This commit is contained in:
ducoterra
2026-08-26 18:42:28 -04:00
parent b2d8696741
commit 1925bb66a8
30 changed files with 2673 additions and 110 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.env.example
+11 -4
View File
@@ -40,12 +40,19 @@ BOR_RRF_K=60 # Reciprocal Rank Fusion damping constant
# BOR_IMPORT_EXTENSIONS=md,markdown,txt,yaml,yml,json,py
# BOR_SUGGESTIONS=["How is my Kubernetes cluster set up?"] # JSON list of onboarding chips
# --- Import sources (git; phase 28) ---
# --- Import sources (git; phase 28, admin-managed since phase 35) ---
# Comma-separated git repo URLs; import_docs clones each (first run) or
# pulls it (subsequent runs) into BOR_SOURCES_DIR/<repo-name>/ and indexes
# the result. Empty = no git sources (import_docs falls back to --source /
# the old ~/Homelab + ~/Deployments defaults). Auth via URL (e.g. an
# https token) or SSH keys.
# the result. Auth via URL (e.g. an https token) or SSH keys.
#
# Phase 35 (owner permission 2026-08-26): the PRIMARY way to manage the
# list is the admin Git sources page (http://localhost:8000/git-sources.html)
# — rows stored in Postgres (git_sources table, migration 0006). This
# variable is the EMPTY-TABLE FALLBACK: it only applies while the admin
# list is empty; once the page has stored any source, this variable is
# ignored (the page is the source of truth). Empty table + empty variable
# = no git sources (import_docs falls back to --source / the old
# ~/Homelab + ~/Deployments defaults; the UI Sync button fails loudly).
# BOR_GIT_SOURCES=https://github.com/user/homelab.git,https://github.com/user/deployments.git
# BOR_SOURCES_DIR=~/bor-sources
Containerfile
+2 -1
View File
@@ -19,9 +19,10 @@ RUN mkdir -p /out/assets \
&& esbuild ./assets/document.js --bundle --minify --format=esm --target=es2022 --outfile=/out/assets/document.js \
&& esbuild ./assets/login.js --bundle --minify --format=esm --target=es2022 --outfile=/out/assets/login.js \
&& esbuild ./assets/tuning.js --bundle --minify --format=esm --target=es2022 --outfile=/out/assets/tuning.js \
&& esbuild ./assets/git-sources.js --bundle --minify --format=esm --target=es2022 --outfile=/out/assets/git-sources.js \
&& esbuild ./assets/markdown.js --minify --outfile=/out/assets/markdown.js \
&& esbuild ./assets/styles.css --minify --outfile=/out/assets/styles.css \
&& cp ./index.html ./sources.html ./document.html ./login.html ./tuning.html /out/
&& cp ./index.html ./sources.html ./document.html ./login.html ./tuning.html ./git-sources.html /out/
# ---------- Stage 2: python dependencies ----------
FROM docker.io/python:3.12-slim AS python
README.md
+49 -23
View File
@@ -65,16 +65,21 @@ uv run python -m scripts.import_docs # import the configured sources (below)
Two source modes:
- **Git sources (recommended)** — set `BOR_GIT_SOURCES` in `.env` to a
comma-separated list of git repo URLs. `import_docs` clones each repo
- **Git sources (recommended)** — the list is managed on the **admin
Git sources page** (`/git-sources.html`) and stored in Postgres (see
[Git-based sources](#git-based-sources)). `import_docs` clones each repo
(first run) or pulls it (subsequent runs) into
`BOR_SOURCES_DIR/<repo-name>/` (default `~/bor-sources`) and indexes the
checkouts — see [Git-based sources](#git-based-sources).
checkouts. While the stored list is empty, the `BOR_GIT_SOURCES` variable
in `.env` is the fallback — the moment the page stores a source, the
variable is ignored.
- **Manual directories** — `--source <path>` (repeatable) imports local
directories directly and *always wins* over `BOR_GIT_SOURCES`.
- If neither is set, the import falls back to the **previous** default,
directories directly and *always wins* over the git sources (stored list
or env).
- If neither is set (stored list, `--source`, and `BOR_GIT_SOURCES` all
empty), the import falls back to the **previous** default,
`~/Homelab` + `~/Deployments` — kept only for backwards compatibility,
now replaced by `BOR_GIT_SOURCES`.
now replaced by the git sources list.
### 6. Run the app
```bash
@@ -102,6 +107,12 @@ uv run uvicorn app.main:app --reload
new tab). **Admin-only** — anonymous visitors see a sign-in gate instead
(the catalog is what the login locks; the document viewer itself stays
open to everyone).
- **Git sources** (`/git-sources.html`) — the list of git repositories the
**Sync sources** button clones and indexes; **admin-only** (the same
sign-in gate as Sources). Add or remove repositories here — no `.env`
editing, no restart. Adding/removing does not clone or prune on its
own: the Sync button performs that, and a removed repository's documents
leave the index on the next sync.
## Thinking
@@ -262,26 +273,38 @@ embedded.
Rather than pointing the import at local folders, point it at **git
repositories** — the notes live in the repos and `import_docs` keeps local
checkouts of them up to date for you:
checkouts of them up to date for you.
**Where the list lives (phase 35):** the primary management surface is
the **admin Git sources page** (`/git-sources.html`) — add or remove
repositories there and the list is stored in Postgres (the `git_sources`
table). `BOR_GIT_SOURCES` in `.env` is the **empty-table fallback**: it
only applies while the stored list is empty, and is ignored once the page
has any row (the page becomes the source of truth — no `.env` editing, no
restart needed afterwards).
```env
# .env
# .env — the fallback list (fresh setups, or until the admin page
# stores a source; phase 35 demotes this variable, it does not remove it)
BOR_GIT_SOURCES=https://git.reeseapps.com/reese/homelab.git,git@github.com:reese/deployments.git
BOR_SOURCES_DIR=~/bor-sources # default; each repo lands in <dir>/<repo-name>/
```
- `BOR_GIT_SOURCES` is a **comma-separated list** of URLs. Auth is whatever
the machine supplies — `https://…` via the OS credential helper, or
`git@host:repo.git` via your SSH key; no credentials are stored in the
app or `.env`.
- The effective list (stored rows, else `BOR_GIT_SOURCES` while the stored
list is empty) is a set of git repo URLs. Auth is whatever the machine
supplies — `https://…` via the OS credential helper, or `git@host:repo.git`
via your SSH key; no credentials are stored in the app or `.env`.
Stored URLs are shape-validated on the page (`https://`, `ssh://`,
`git@…` — scp-style `host:repo` is rejected).
- Every run **clones** each repo (first time, shallow `--depth 1`) or
**pulls** it (`git pull --ff-only` — fast-forward only, so a diverged or
broken checkout fails loudly instead of merging) into
`BOR_SOURCES_DIR/<repo-name>/`, then indexes the checkouts exactly like
any local directory (A9 format filter, hidden-dir skip, sha256 delta).
`documents.source` is the repo directory name (e.g. `homelab`).
- **`--source <path>` overrides**: when the flag is given,
`BOR_GIT_SOURCES` is ignored and the manual directory(ies) are imported.
- **`--source <path>` overrides**: when the flag is given, the git
sources (stored list *and* `BOR_GIT_SOURCES`) are ignored and the
manual directory(ies) are imported.
- **A failed sync aborts the run**: if any repo cannot be cloned/pulled,
`import_docs` exits non-zero naming the failing repo and imports
**nothing** (no partial junk). Fix the URL/connectivity and re-run — the
@@ -293,9 +316,10 @@ The **Sync sources** button on the **Sources** page — visible to the
**admin only** (anonymous visitors never see it) — runs the whole
git-source refresh in one click, in-process:
1. **clone/pull** every `BOR_GIT_SOURCES` repo (the same
`clone_or_pull` the CLI uses — shallow clone on first run,
`git pull --ff-only` afterwards);
1. **clone/pull** every configured git source — the admin-managed list
(the `git_sources` table; `BOR_GIT_SOURCES` only while that list is
empty), through the same `clone_or_pull` the CLI uses (shallow clone
on first run, `git pull --ff-only` afterwards);
2. **re-import with prune** — the `--prune` equivalent, so files deleted
upstream leave the index (the button is the canonical "mirror the
repos" action); the sha256 delta still skips unchanged files, so an
@@ -304,10 +328,12 @@ git-source refresh in one click, in-process:
chat turn injects) — but only when the import actually changed the
knowledge base.
- **Prerequisites:** `BOR_GIT_SOURCES` must be set — an unset/empty list
fails the sync loudly ("no git sources configured"), because the button
targets the git repos only (manual `--source` directories have no repo
to clone) — and `git` must be on the app's `PATH`.
- **Prerequisites:** at least one git source must be configured — a row
on the admin Git sources page, or `BOR_GIT_SOURCES` in `.env` while the
stored list is empty; **both** empty fails the sync loudly ("no git
sources configured"), because the button targets the git repos only
(manual `--source` directories have no repo to clone) — and `git` must
be on the app's `PATH`.
- **States:** clicking starts the run (`202`) and the button goes
disabled with **Syncing…** (spinning icon) while the page polls
`GET /api/sync/status` every 2 s. There is deliberately **no
@@ -534,8 +560,8 @@ served locally (no CDN), `BOR_ENVIRONMENT=production`.
| `BOR_HYBRID_LEXICAL_CANDIDATES` | `30` | FTS list width for the RRF fusion |
| `BOR_RRF_K` | `60` | RRF damping constant (`1/(k + rank)`) |
| `BOR_IMPORT_EXTENSIONS` | `md,markdown,txt,yaml,yml,json,py` | csv of importable formats (may only narrow the A9 set) |
| `BOR_GIT_SOURCES` | — (empty) | csv of git repo URLs; `import_docs` clones/pulls them into `BOR_SOURCES_DIR` and indexes the checkouts (see *Git-based sources*) |
| `BOR_SOURCES_DIR` | `~/bor-sources` | where the `BOR_GIT_SOURCES` repos are cloned/pulled (one subdirectory per repo) |
| `BOR_GIT_SOURCES` | — (empty) | csv of git repo URLs — **fallback while the admin Git sources page's list (Postgres `git_sources`) is empty**; the page is the primary management surface (see *Git-based sources*) |
| `BOR_SOURCES_DIR` | `~/bor-sources` | where the git source repos are cloned/pulled (one subdirectory per repo) |
| `BOR_STEERING_MAX_CHARS` | `8000` | char budget for the `<tuning>` (steering notes) prompt section |
| `BOR_SUMMARY_MAX_CHARS` | `12000` | cap on document content sent to the `lite` summary model at import (see *Document summaries*) |
| `BOR_KB_OVERVIEW_MAX_CHARS` | `4000` | char budget for the `<knowledge_base>` (KB overview) prompt section |
alembic/versions/0006_git_sources.py
+48
View File
@@ -0,0 +1,48 @@
"""git sources: admin-managed list of git repo URLs
Revision ID: 0006
Revises: 0005
Create Date: 2026-08-26
Phase 35 (git-sources-admin story, A13 — one additive, reversible table,
no other schema change):
* ``git_sources`` — one row per admin-managed git source URL. The admin
page (``/git-sources.html``) maintains the list that the Sync button
(phase 32) and ``import_docs`` (phase 28) clone/pull. DB rows win over
the ``BOR_GIT_SOURCES`` env var, which stays a fallback while the table
is empty (once the table has rows it is ignored — the UI is the source
of truth). ``url`` is unique (``uq_git_sources_url``); ``added_at`` is
stamped server-side and orders the list oldest-first.
"""
from __future__ import annotations
import sqlalchemy as sa
from sqlalchemy.dialects import postgresql
from alembic import op
revision = "0006"
down_revision = "0005"
branch_labels = None
depends_on = None
def upgrade() -> None:
op.create_table(
"git_sources",
sa.Column("id", postgresql.UUID(as_uuid=True), primary_key=True),
sa.Column("url", sa.Text(), nullable=False),
sa.Column(
"added_at",
sa.DateTime(timezone=True),
server_default=sa.func.now(),
nullable=False,
),
)
op.create_index("uq_git_sources_url", "git_sources", ["url"], unique=True)
def downgrade() -> None:
op.drop_index("uq_git_sources_url", table_name="git_sources")
op.drop_table("git_sources")
app/api/git_sources.py
+132
View File
@@ -0,0 +1,132 @@
"""Admin-managed git sources API (phase 35, task 02).
Admin-only CRUD under ``/api/git-sources`` (phase 16 pattern, A10
extended — the public API surface stays stateless and the signed cookie
remains the only session state, same as ``/api/steering`` and
``/api/sync``): the ``git_sources`` table holds the repo URLs the Sync
button (phase 32) and ``import_docs`` (phase 28) clone/pull. DB rows win
over ``BOR_GIT_SOURCES``, which is a fallback while the table is empty
(the phase's locked decision — ``from_env`` tells the UI which list it is
looking at, so the page can show the env note only while the fallback is
active).
Routes: ``GET`` (DB rows oldest-first, or the env list with
``from_env: true`` while the table is empty), ``POST`` (201, validated
create), ``DELETE /{source_id}`` (204). The whole router sits behind
:func:`app.core.auth.require_admin` — anonymous callers get 403 on every
route.
No credential-echo path: URLs may embed ``user:pass@`` (phase 32's
masking discipline), so the 409/422 details are fixed generic strings
that never repeat the submitted URL.
Scope boundary (phase locked decisions): adding or removing a repo does
NOT clone, import, or prune anything — the existing Sync button performs
that (a removal prunes on the next sync, ``prune=True``).
"""
from __future__ import annotations
import re
import uuid
from fastapi import APIRouter, Depends, HTTPException, Response
from sqlalchemy import select
from sqlalchemy.exc import IntegrityError
from sqlalchemy.orm import Session
from app.config import get_settings
from app.core.auth import require_admin
from app.db import get_db
from app.models import GitSource
from app.schemas import GitSourceIn, GitSourceList, GitSourceOut
router = APIRouter(
prefix="/git-sources",
tags=["git-sources"],
dependencies=[Depends(require_admin)], # phase 16 pattern: admin-only surface
)
#: Accepted git URL shapes — the trimmed URL must *start* with one of them.
#: Covers the phase-28 real URLs (HTTPS + ``git@`` SSH); scp-style
#: ``host:repo`` is deliberately rejected (422). ASSUMPTION (task 02): the
#: accepted shapes are exactly these four prefixes.
URL_RE = re.compile(r"^(https?://|ssh://|git@)")
@router.get("", response_model=GitSourceList)
def list_git_sources(
db: Session = Depends(get_db), # noqa: B008
) -> GitSourceList:
"""The effective git source list.
DB rows ordered by ``(added_at, id)`` (oldest first, id tie-break for
same-timestamp inserts) with ``from_env: false``; while the table is
empty, the ``BOR_GIT_SOURCES`` env URLs as rows with null
``id``/``added_at`` and ``from_env: true``.
"""
rows = db.scalars(
select(GitSource).order_by(GitSource.added_at.asc(), GitSource.id.asc())
).all()
if rows:
return GitSourceList(
sources=[GitSourceOut(id=row.id, url=row.url, added_at=row.added_at) for row in rows],
from_env=False,
)
return GitSourceList(
sources=[
GitSourceOut(id=None, url=url, added_at=None)
for url in get_settings().git_source_list
],
from_env=True,
)
@router.post("", response_model=GitSourceOut, status_code=201)
def create_git_source(
payload: GitSourceIn,
db: Session = Depends(get_db), # noqa: B008
) -> GitSourceOut:
"""Store one repo URL (already trimmed by the schema).
422 when the shape is not one of the accepted prefixes (generic
detail — the input is never echoed); 409 when the trimmed URL is
already stored (same; the unique index is the backstop against a
concurrent insert the pre-check missed); 201 + the created row
otherwise.
"""
url = payload.url
if not URL_RE.match(url):
raise HTTPException(
status_code=422, detail="not a valid git URL (expected https://, ssh:// or git@…)"
)
if db.scalar(select(GitSource).where(GitSource.url == url)) is not None:
raise HTTPException(status_code=409, detail="a git source with this URL already exists")
row = GitSource(url=url)
db.add(row)
try:
db.commit()
except IntegrityError:
db.rollback()
raise HTTPException(
status_code=409, detail="a git source with this URL already exists"
) from None
db.refresh(row)
return GitSourceOut(id=row.id, url=row.url, added_at=row.added_at)
@router.delete("/{source_id}", status_code=204)
def delete_git_source(
source_id: uuid.UUID,
db: Session = Depends(get_db), # noqa: B008
) -> Response:
"""Remove a stored row; 404 when the id is unknown.
Removing does not touch the clones or the index — the next Sync
(``prune=True``) prunes the dropped repo (phase scope boundary).
"""
row = db.get(GitSource, source_id)
if row is None:
raise HTTPException(status_code=404, detail="git source not found")
db.delete(row)
db.commit()
return Response(status_code=204)
app/api/sync.py
+24 -9
View File
@@ -15,9 +15,12 @@ stale button state (§7.4 adaptation, phase locked decisions).
Pipeline (the canonical "mirror the repos" action — phase locked
decisions):
1. resolve the ``BOR_GIT_SOURCES`` URLs — empty/missing fails loudly
(``no git sources configured``) instead of silently importing the
legacy local directories;
1. resolve the effective git sources — the ``git_sources`` DB rows,
else the ``BOR_GIT_SOURCES`` fallback
(:func:`app.rag.git_sources.effective_git_sources`, shared with the
CLI) — empty on both origins fails loudly (``no git sources
configured``) instead of silently importing the legacy local
directories;
2. :func:`scripts.git_sync.clone_or_pull` each repo into
``BOR_SOURCES_DIR/<repo-name>/`` (phase 28 — reused, not
re-implemented; a failing repo aborts before any import);
@@ -44,6 +47,8 @@ from fastapi import APIRouter, Depends, HTTPException
from app.config import get_settings
from app.core.auth import require_admin
from app.db import SessionLocal
from app.rag.git_sources import effective_git_sources
from app.rag.importer import ImportSummary, import_sources
from app.rag.llm import LLMClient
from app.rag.overview import regenerate_overview
@@ -141,13 +146,23 @@ async def _run_sync() -> None:
_status.error = None
try:
settings = get_settings()
git_urls = settings.git_source_list
# The background task has no request session: open a short-lived
# one around the shared phase-35 resolver (DB rows win, the
# BOR_GIT_SOURCES list is a fallback while the table is empty).
db = SessionLocal()
try:
git_urls, origin = effective_git_sources(db)
finally:
db.close()
if not git_urls:
# The button targets BOR_GIT_SOURCES only (manual --source
# dirs have no repo to clone) — an empty config fails loudly
# instead of silently importing the legacy directories.
raise GitSyncError("no git sources configured (BOR_GIT_SOURCES)")
logger.info("sync: started repos=%d", len(git_urls))
# The button targets git sources only (manual --source dirs
# have no repo to clone) — an empty config on *both* origins
# fails loudly instead of silently importing the legacy
# directories.
raise GitSyncError(
"no git sources configured (git_sources table empty and BOR_GIT_SOURCES unset)"
)
logger.info("sync: started repos=%d origin=%s", len(git_urls), origin)
sources_root = Path(settings.sources_dir).expanduser()
sources = [clone_or_pull(url, sources_root / repo_name(url)) for url in git_urls]
llm = LLMClient()
app/core/caching.py
+5 -1
View File
@@ -111,7 +111,10 @@ def asset_version(static_dir: str | None = None) -> str:
# Response-layer cache busting (task 02)
# ---------------------------------------------------------------------------
#: The five known HTML pages — the ONLY paths whose body is rewritten.
#: The known HTML pages — the ONLY paths whose body is rewritten.
#: Phase 35 adds the admin git sources page: without this entry it
#: would serve unversioned asset refs, which the immutable-for-a-year
#: asset caching would pin to stale CSS after a deploy.
HTML_PAGES: tuple[str, ...] = (
"/",
"/index.html",
@@ -119,6 +122,7 @@ HTML_PAGES: tuple[str, ...] = (
"/document.html",
"/login.html",
"/tuning.html",
"/git-sources.html", # phase 35: the admin git sources page
)
#: Prefix of the versioned static assets (header-only caching; the body is
app/main.py
+2
View File
@@ -22,6 +22,7 @@ from starlette.middleware.sessions import SessionMiddleware
from app.api.auth import router as auth_router
from app.api.chat import router as chat_router
from app.api.docs import router as docs_router
from app.api.git_sources import router as git_sources_router
from app.api.health import router as health_router
from app.api.steering import router as steering_router
from app.api.suggestions import router as suggestions_router
@@ -64,6 +65,7 @@ def create_app() -> FastAPI:
app.include_router(auth_router, prefix="/api")
app.include_router(suggestions_router, prefix="/api")
app.include_router(docs_router, prefix="/api")
app.include_router(git_sources_router, prefix="/api")
app.include_router(chat_router, prefix="/api")
app.include_router(steering_router, prefix="/api")
app.include_router(sync_router, prefix="/api")
app/models.py
+18
View File
@@ -13,6 +13,8 @@ Data model — see ``.agent/PLAN.md`` §Data Model:
* ``kb_overview`` — single-row lite-generated outline of the KB's basic
categories, injected as the ``<knowledge_base>``
section of every chat turn (phase 31).
* ``git_sources`` — admin-managed git source URLs the Sync button and
import_docs clone/pull (phase 35).
"""
from __future__ import annotations
@@ -133,3 +135,19 @@ class KbOverview(Base):
updated_at: Mapped[datetime] = mapped_column(
DateTime(timezone=True), server_default=func.now()
)
class GitSource(Base):
"""One admin-managed git source (phase 35).
The UI-maintained list of repo URLs the Sync button (phase 32) and
import_docs (phase 28) clone/pull. DB rows win over the
BOR_GIT_SOURCES env var, which is a fallback while this table is
empty (see app.rag.git_sources.effective_git_sources).
"""
__tablename__ = "git_sources"
id: Mapped[uuid.UUID] = mapped_column(UUID(as_uuid=True), primary_key=True, default=uuid.uuid4)
url: Mapped[str] = mapped_column(Text, unique=True, nullable=False)
added_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), server_default=func.now())
app/rag/git_sources.py
+44
View File
@@ -0,0 +1,44 @@
"""The shared git-source resolver (phase 35, task 03).
One function, two callers: the in-app Sync pipeline
(``app/api/sync.py::_run_sync``) and the CLI
(``scripts/import_docs.py::_resolve_sources``) — both resolve the repo
URLs to clone/pull through :func:`effective_git_sources`, so the
admin-managed ``git_sources`` table is what actually gets cloned and
indexed from either entry point (the API's ``GET /api/git-sources``
fallback list is the only other place that reads the env list — task 02).
Precedence (the phase's locked decision — the env var is demoted, not
removed): ``git_sources`` DB rows win, in ``(added_at, id)`` order;
``BOR_GIT_SOURCES`` is a fallback only while the table is empty; both
empty → ``([], "env")`` and each caller keeps its existing fail-loud
behavior (sync: ``GitSyncError``; CLI: the legacy ``DEFAULT_SOURCES``
fallback).
"""
from __future__ import annotations
from typing import Literal
from sqlalchemy import select
from sqlalchemy.orm import Session
from app.config import get_settings
from app.models import GitSource
def effective_git_sources(db: Session) -> tuple[list[str], Literal["db", "env"]]:
"""``(urls, origin)`` — the repo URLs to clone, and where they came from.
``"db"``: the ``git_sources`` rows in ``(added_at, id)`` order (oldest
first, id tie-break for same-timestamp inserts) — once the table has
any row, ``BOR_GIT_SOURCES`` is ignored entirely.
``"env"``: the ``BOR_GIT_SOURCES`` list — ``Settings.git_source_list``,
the phase-28 CSV parse, reused not re-implemented — used only while
the table is empty; both empty → ``([], "env")``.
"""
rows = db.scalars(
select(GitSource).order_by(GitSource.added_at.asc(), GitSource.id.asc())
).all()
if rows:
return [row.url for row in rows], "db"
return list(get_settings().git_source_list), "env"
app/schemas.py
+43
View File
@@ -154,3 +154,46 @@ class SteeringNoteList(BaseModel):
"""``GET /api/steering`` response: all notes, newest first."""
notes: list[SteeringNote]
class GitSourceIn(BaseModel):
"""``POST /api/git-sources`` body: one repo URL (phase 35, task 02).
Mirrors :class:`SteeringNoteIn` — the URL is trimmed *before* the
length constraints run, so a whitespace-only body is a 422 and a URL
with surrounding spaces is stored clean. Shape validation
(``https://``, ``ssh://``, ``git@``) happens in the API layer so the
422 detail can be one generic string that never echoes the input.
"""
url: str = Field(min_length=1, max_length=500)
@field_validator("url", mode="before")
@classmethod
def _trim_url(cls, v: object) -> object:
return v.strip() if isinstance(v, str) else v
class GitSourceOut(BaseModel):
"""One git source as returned by the API (phase 35, task 02).
``id`` / ``added_at`` are nullable: env-fallback rows (table empty →
the list comes from ``BOR_GIT_SOURCES``) carry neither, only a URL.
"""
id: uuid.UUID | None
url: str
added_at: datetime | None
class GitSourceList(BaseModel):
"""``GET /api/git-sources`` response (phase 35, task 02).
``from_env`` is True only when the ``git_sources`` table is empty and
the list comes from ``BOR_GIT_SOURCES`` (the phase's locked fallback);
once the table has rows the env var is ignored and ``from_env`` is
False — the UI is the source of truth.
"""
sources: list[GitSourceOut]
from_env: bool
frontend/assets/git-sources.js
+320
View File
@@ -0,0 +1,320 @@
/* Brain of Reese — Git sources admin page (phase 35, task 04).
*
* The page module for /git-sources.html: the admin-only manager for the
* stored git source list (git-sources table, phase 35 tasks 01/02).
* This module is the single owner of the page's behaviour:
*
* • boot — initSharedHeader() (one cached whoami, shared with the
* header toggling): anonymous → the sign-in gate shows and the
* manager stays hidden (the exact Sources page gate pattern, and
* NO /api/git-sources call is made); admin → gate hidden,
* #git-sources-content revealed, loadSources().
* • loadSources() — GET /api/git-sources → the table rows
* (#git-sources-tbody), the env-fallback note's visibility
* (from_env), and the empty state. URLs are ALWAYS rendered with
* textContent — never innerHTML (they may embed user:pass@
* credentials; phase 32's masking discipline). Non-2xx or a
* network failure renders the role="alert" load error with a
* retry button — never a stuck page.
* • add — #git-source-form submit → POST /api/git-sources {url}.
* §7.4 never-stale: the button disables + relabels "Adding…"
* while the request is out, re-enables ("Add source") on success
* AND failure. 201 clears the input, reloads the list, and
* focuses the new row's Remove button (a11y); a failure (409
* duplicate, 422 shape) shows the server detail inline under the
* form (role="alert", 422 shape-aware like the tuning forms) and
* keeps the input — the instruction survives.
* • remove — a row's Remove button asks window.confirm first
* (removal prunes the documents only on the NEXT sync — the
* confirm says so). Cancel → nothing; ok → the row button
* disables, DELETE /api/git-sources/{id}, loadSources(). A
* failure shows a per-row role="alert" error and re-enables the
* button. Env-fallback rows (id null — the list comes from
* BOR_GIT_SOURCES, not the table) carry no Remove: nothing is
* stored to remove — they show a "from .env" tag instead.
* • announce(msg) — #git-sources-announcer (role=status,
* aria-live=polite): the screen-reader confirmation for loads,
* adds, and removals.
*
* Scope boundary (phase locked decisions): adding or removing a repo
* does NOT clone, import, or prune — the header's Sync sources button
* (module-owned in assets/header.js) performs that; the page's hint
* box says so.
*
* The shared header module loads through this script's own relative
* import ("./header.js") — a hoisted import evaluated before this body
* runs (single-evaluation design: no direct <script> tag; esbuild
* inlines it into the page bundle in the image build).
*/
import { initSharedHeader } from "./header.js";
/* ---------- page elements (git-sources.html, task 04) ---------- */
const gateEl = document.querySelector("#git-sources-gate");
const contentEl = document.querySelector("#git-sources-content");
const formEl = document.querySelector("#git-source-form");
const urlInput = document.querySelector("#git-source-url");
const addBtn = document.querySelector("#git-source-add");
const addError = document.querySelector("#git-source-error");
const loadErrorEl = document.querySelector("#git-sources-load-error");
const loadErrorText = document.querySelector("#git-sources-load-error-text");
const retryBtn = document.querySelector("#git-sources-retry");
const tableWrap = document.querySelector("#git-sources-table-wrap");
const tbody = document.querySelector("#git-sources-tbody");
const emptyEl = document.querySelector("#git-sources-empty");
const envNote = document.querySelector("#git-sources-env-note");
const announcer = document.querySelector("#git-sources-announcer");
/* Polite live region: the screen-reader confirmation for loads, adds,
and removals (the phase-15 announcer pattern). */
function announce(message) {
if (announcer) announcer.textContent = message;
}
/* Added date — localized (toLocaleString); env-fallback rows carry
added_at null, and a corrupt timestamp must not blank the cell. */
function fmtDate(iso) {
if (!iso) return "—";
try {
return new Date(iso).toLocaleString();
} catch {
return "—";
}
}
/* FastAPI error bodies: a string detail or the validation-error array
(the first entry's msg is the human line). Same extraction as
tuning.js — 422 shape-aware. */
async function apiDetail(r, fallback) {
try {
const data = await r.json();
if (Array.isArray(data.detail) && data.detail[0] && data.detail[0].msg) {
return String(data.detail[0].msg);
}
if (typeof data.detail === "string" && data.detail) return data.detail;
} catch {
/* non-JSON error body */
}
return fallback;
}
/* ---------- load / render ---------- */
/* GET /api/git-sources → the table rows + env note + empty state.
The load error (role="alert" + retry) is the ONLY terminal state a
failed fetch may reach — never a stuck page. */
async function loadSources() {
let r;
try {
r = await fetch("/api/git-sources");
} catch {
showLoadError("Could not reach the server — is the app running?");
return;
}
if (!r.ok) {
showLoadError(await apiDetail(r, `The server could not list the git sources (${r.status}).`));
return;
}
let data;
try {
data = await r.json();
} catch {
showLoadError("The server sent an unreadable list — try again.");
return;
}
hideLoadError();
const sources = Array.isArray(data.sources) ? data.sources : [];
renderSources(sources, data.from_env === true);
announce(`${sources.length} git source${sources.length === 1 ? "" : "s"} listed.`);
}
function showLoadError(message) {
if (loadErrorText) loadErrorText.textContent = message;
if (loadErrorEl) loadErrorEl.hidden = false;
// The list state is unknown — hide the table AND the empty state so
// the error is the only claim about the list's contents.
if (tableWrap) tableWrap.hidden = true;
if (emptyEl) emptyEl.hidden = true;
}
function hideLoadError() {
if (loadErrorEl) loadErrorEl.hidden = true;
if (loadErrorText) loadErrorText.textContent = "";
}
function renderSources(sources, fromEnv) {
if (envNote) envNote.hidden = !fromEnv;
if (tbody) {
tbody.replaceChildren();
for (const s of sources) tbody.appendChild(makeRow(s));
}
const hasRows = sources.length > 0;
if (tableWrap) tableWrap.hidden = !hasRows;
if (emptyEl) emptyEl.hidden = hasRows;
}
/* One row: the URL in a mono <code> (textContent only — URLs may
contain credentials), the localized added date ("—" for env
fallback rows), and the per-row Remove button — or the "from .env"
tag for env-fallback rows (id null: nothing is stored to remove;
the env note says where the active list comes from). */
const REMOVE_ICON =
'<svg aria-hidden="true" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="1.8" stroke-linecap="round"><path d="M5 7h14M10 7V5h4v2M8.5 7l.7 12h5.6l.7-12"/></svg>';
function makeRow(s) {
const tr = document.createElement("tr");
if (s.id) tr.dataset.id = s.id;
const urlTd = document.createElement("td");
urlTd.className = "git-source-url-cell";
urlTd.title = s.url; // full URL on hover (long URLs scroll the wrapper)
const code = document.createElement("code");
code.textContent = s.url; // rendered as text, never as HTML
urlTd.appendChild(code);
tr.appendChild(urlTd);
const addedTd = document.createElement("td");
addedTd.textContent = fmtDate(s.added_at);
tr.appendChild(addedTd);
const actTd = document.createElement("td");
actTd.className = "git-source-actions-cell";
if (s.id) {
const btn = document.createElement("button");
btn.type = "button";
btn.className = "git-source-remove";
btn.setAttribute("aria-label", `Remove git source: ${s.url}`);
btn.innerHTML = REMOVE_ICON + "<span>Remove</span>";
const rowError = document.createElement("span");
rowError.className = "git-source-row-error";
rowError.setAttribute("role", "alert");
rowError.hidden = true;
btn.addEventListener("click", () => removeSource(s, btn, rowError));
actTd.append(btn, rowError);
} else {
const tag = document.createElement("span");
tag.className = "git-source-env-tag";
tag.textContent = "from .env";
actTd.appendChild(tag);
}
tr.appendChild(actTd);
return tr;
}
/* ---------- remove (DELETE /api/git-sources/{id}) ----------
* Removal does not prune anything immediately — the next sync does
* (phase scope boundary), so the confirm says exactly that. Cancel →
* nothing; a failed delete → per-row role="alert" error + re-enabled
* button (never a stuck row); success → the list reloads. */
async function removeSource(s, btn, rowError) {
const ok = window.confirm(
"Remove this git source from the list? Its documents stay indexed until the next sync prunes them.",
);
if (!ok) return;
btn.disabled = true; // one delete per click
rowError.hidden = true;
try {
const r = await fetch(`/api/git-sources/${encodeURIComponent(s.id)}`, { method: "DELETE" });
if (!r.ok) {
rowError.textContent = await apiDetail(r, "Could not remove the git source — try again.");
rowError.hidden = false;
btn.disabled = false;
return;
}
announce("Git source removed.");
await loadSources(); // 204: the server confirmed — the list re-renders
} catch {
rowError.textContent = "Could not remove the git source — is the app reachable?";
rowError.hidden = false;
btn.disabled = false;
}
}
/* ---------- add (POST /api/git-sources) ---------- */
if (formEl && urlInput && addBtn) {
formEl.addEventListener("submit", async (e) => {
e.preventDefault();
// Client-side non-empty check (the input is `required` too — the
// browser's native prompt is the first line, this one the second).
const url = urlInput.value.trim();
if (!url) {
if (addError) {
addError.textContent = "Enter a git URL to add.";
addError.hidden = false;
}
return;
}
if (addError) addError.hidden = true; // a new attempt starts clean
addBtn.disabled = true; // §7.4: one POST per click
addBtn.textContent = "Adding…";
try {
const r = await fetch("/api/git-sources", {
method: "POST",
headers: { "Content-Type": "application/json" },
body: JSON.stringify({ url }),
});
if (r.ok) {
let createdId = null;
try {
createdId = (await r.json()).id ?? null;
} catch {
/* the 201 body is advisory — the reload is the truth */
}
urlInput.value = ""; // 201: the source is stored
announce("Git source added.");
await loadSources(); // the new row lands in the table
focusNewRow(createdId); // a11y: land the caret on the new row
return;
}
// 409 duplicate / 422 shape / anything else: the server detail
// inline (never echoing a URL the server wouldn't), form kept —
// the input survives so the fix is one edit, not a re-type.
if (addError) {
addError.textContent = await apiDetail(r, "Could not add the git source — try again.");
addError.hidden = false;
}
} catch {
if (addError) {
addError.textContent = "Could not add the git source — is the app reachable?";
addError.hidden = false;
}
} finally {
addBtn.disabled = false; // never stale — success OR failure
addBtn.textContent = "Add source";
}
});
}
/* After a successful add, focus the new row's Remove button so the
keyboard/screen-reader user lands where the new data is. The 201
body carries the row id (tr[data-id]); without one, the first row
is the fallback (the list is small and ordered). */
function focusNewRow(createdId) {
if (!tbody) return;
const row = createdId
? tbody.querySelector(`tr[data-id="${CSS.escape(createdId)}"]`)
: null;
const target = (row || tbody.querySelector("tr"))?.querySelector(".git-source-remove");
if (target) target.focus();
}
/* ---------- retry + boot ---------- */
if (retryBtn) retryBtn.addEventListener("click", () => loadSources());
(async () => {
// The shared header FIRST (Sign in/out + the admin-only nav links +
// the Sync button — one cached whoami), then the gate: anonymous
// visitors get the gate and NO /api/git-sources call (the Sources
// page gate pattern); the admin gets the manager.
const admin = await initSharedHeader();
if (!admin) {
if (gateEl) gateEl.hidden = false;
if (contentEl) contentEl.hidden = true; // ships hidden — stays hidden
return;
}
if (gateEl) gateEl.hidden = true;
if (contentEl) contentEl.hidden = false;
await loadSources();
})();
frontend/assets/header.js
+8 -2
View File
@@ -6,8 +6,9 @@
*
* • the Sign in / Sign out auth pair (phase 16, exactly one visible —
* decided by /api/whoami at load);
* • the admin-only nav links — "Sources" (#nav-sources, phase 19)
* and "Tuning" (#nav-tuning, phase 29) — phase 19 UX revision
* • the admin-only nav links — "Sources" (#nav-sources, phase 19),
* "Git sources" (#nav-git-sources, phase 35) and "Tuning"
* (#nav-tuning, phase 29) — phase 19 UX revision
* (owner permission 2026-08-23): hidden for anonymous on EVERY
* page, revealed for admin. Phase 34 task 03 (owner confirmation
* 2026-08-26): the SAME nav ships on all five pages (chat,
@@ -109,6 +110,11 @@ export async function initSharedHeader() {
if (signOut) signOut.hidden = !admin;
const navSources = document.querySelector("#nav-sources");
if (navSources) navSources.hidden = !admin;
// Phase 35 (owner permission 2026-08-26): the Git sources nav link —
// admin-only, the same ship-hidden / reveal-for-admin contract as
// the Sources link above.
const navGitSources = document.querySelector("#nav-git-sources");
if (navGitSources) navGitSources.hidden = !admin;
// Phase 29: the Global Tuning nav link (every page from phase 34
// task 03) — admin-only, the same ship-hidden / reveal-for-admin
// contract as the Sources link.
frontend/assets/styles.css
+274 -11
View File
@@ -1275,6 +1275,253 @@ details.thinking .thinking-text ul { margin: 0 0 0.5rem; }
.docs-table tbody tr:hover { background: var(--bg); }
.docs-table tbody tr:last-child td { border-bottom: 0; }
/* ---------- Git sources page (phase 35) ----------
/git-sources.html: the admin-only manager for the stored git source
list (add / remove, git-sources table). Same full-width table
language as the Sources page (no skinny single-column list), the
phase-16 gate reused verbatim (.sources-gate classes), and the
phase-27 form-card language for the add form. Every pair reuses the
Phase-08 AA palette: dark ink on brand 5.2:1 (never white on
brand, 3.7:1, fails), brand-ink/brand-soft 6.9:1, err 9.1:1,
ink-soft >=6.9:1. Touch targets >=44px; :focus-visible via the
global 3px outline rule. No filter: blur, no CDN, system fonts. */
.git-sources-shell {
display: flex;
flex-direction: column;
gap: 1.25rem;
flex: 1;
}
/* Add form — the tuning form's surface as a single row: visible label
+ mono URL input (the credentials case is real, so the input is
mono) + the brand "Add source" button; wraps to a column at narrow
widths (the <=640px block below). */
#git-source-form {
display: flex;
flex-wrap: wrap;
align-items: center;
gap: 0.6rem;
background: var(--surface);
border: 1px solid var(--line);
border-radius: var(--radius);
box-shadow: var(--shadow);
padding: 0.9rem 1rem 1rem;
}
#git-source-form:focus-within { border-color: var(--brand); box-shadow: 0 0 0 3px var(--brand-soft), var(--shadow); }
#git-source-form > label { color: var(--ink); font-weight: 600; white-space: nowrap; }
#git-source-url {
flex: 1;
min-width: 14rem;
min-height: 44px;
font-family: var(--mono);
font-size: 0.88rem;
color: var(--ink);
background: var(--bg);
border: 1px solid var(--line);
border-radius: var(--radius-sm);
padding: 0.45rem 0.7rem;
}
#git-source-url::placeholder { color: var(--ink-soft); }
#git-source-url:focus-visible { outline-offset: 0; border-color: var(--brand); }
#git-source-add {
display: inline-flex;
align-items: center;
justify-content: center;
min-height: 44px;
padding: 0.4rem 1.2rem;
border: 0;
border-radius: var(--radius-sm);
background: var(--brand);
color: var(--bg); /* dark ink on brand: 5.2:1 */
font: inherit;
font-weight: 700;
cursor: pointer;
}
#git-source-add:hover:not(:disabled) { background: #7d88f5; }
#git-source-add:disabled { opacity: 0.6; cursor: wait; }
/* The add form's inline error (role=alert): the err pair (9.1:1);
flex-basis 100% drops it onto its own row under the input. */
.git-source-error {
flex-basis: 100%;
margin: 0;
background: var(--err-bg);
color: var(--err-ink);
border: 1px solid var(--err-line);
border-radius: var(--radius-sm);
padding: 0.45rem 0.8rem;
font-size: 0.85rem;
font-weight: 600;
}
/* Load failure (role=alert) + the retry action: a failed
GET /api/git-sources must never leave a stuck page. The retry
button keeps the err pair on the err surface (9.1:1). */
.git-source-load-error {
display: flex;
flex-wrap: wrap;
align-items: center;
gap: 0.6rem;
background: var(--err-bg);
color: var(--err-ink);
border: 1px solid var(--err-line);
border-radius: var(--radius);
box-shadow: var(--shadow);
padding: 0.7rem 0.9rem;
font-size: 0.9rem;
font-weight: 600;
}
.git-source-load-error > span { flex: 1; min-width: 12rem; }
#git-sources-retry {
min-height: 44px;
padding: 0.4rem 1rem;
border: 1px solid var(--err-line);
border-radius: var(--radius-sm);
background: transparent;
color: var(--err-ink);
font: inherit;
font-weight: 700;
cursor: pointer;
}
#git-sources-retry:hover { background: rgb(239 68 68 / 0.12); }
/* Env-fallback note (from_env: true — the table is empty and the list
is BOR_GIT_SOURCES): the info chip in the theme palette —
brand-soft surface, brand-ink text (6.9:1). */
.git-source-env-note {
margin: 0;
background: var(--brand-soft);
color: var(--brand-ink);
border: 1px solid var(--brand-soft);
border-radius: var(--radius-sm);
padding: 0.6rem 0.9rem;
font-size: 0.88rem;
}
.git-source-env-note code {
font-family: var(--mono);
font-size: 0.85em;
background: var(--surface);
color: var(--brand-ink); /* 8.7:1 on surface */
padding: 0.1em 0.35em;
border-radius: 5px;
}
/* Hint box (role=note): the page-sub styling family — ink-soft on
surface (6.9:1), a dashed border marks it as guidance, not state.
It names the Sync button: the action that clones the listed repos
and prunes the removed ones (the phase scope boundary). */
.git-source-hint {
margin: 0;
color: var(--ink-soft);
background: var(--surface);
border: 1px dashed var(--line);
border-radius: var(--radius-sm);
padding: 0.6rem 0.9rem;
font-size: 0.88rem;
}
/* The list: the Sources page's table pattern — full width in the
72rem frame, surface card, horizontally scrollable wrapper (the
URL column never wraps or ellipsizes: long URLs, credentials
included, scroll the wrapper instead of truncating). */
#git-sources-table-wrap {
background: var(--surface);
border: 1px solid var(--line);
border-radius: var(--radius);
box-shadow: var(--shadow);
overflow-x: auto;
}
.git-sources-table {
width: 100%;
border-collapse: collapse;
min-width: 560px;
font-size: 0.93rem;
}
.git-sources-table th, .git-sources-table td {
text-align: left;
padding: 0.7rem 1rem;
border-bottom: 1px solid var(--line);
white-space: nowrap;
}
.git-sources-table th {
background: var(--brand-soft);
color: var(--brand-ink);
font-size: 0.82rem;
text-transform: uppercase;
letter-spacing: 0.04em;
position: sticky;
top: 0;
}
/* URL cell: mono at the Sources-table size; the <code> is plain
(no chip background — the cell IS the mono readout). */
.git-sources-table td.git-source-url-cell { font-family: var(--mono); font-size: 0.82rem; }
.git-sources-table td.git-source-url-cell code { font-family: inherit; }
.git-sources-table tbody tr:hover { background: var(--bg); }
.git-sources-table tbody tr:last-child td { border-bottom: 0; }
/* Per-row Remove: the tuning row-action language (icon + label,
>=44px) with the Delete hover pair (err 9.1:1). */
.git-source-remove {
display: inline-flex;
align-items: center;
justify-content: center;
gap: 0.35rem;
min-height: 44px;
min-width: 44px;
flex: 0 0 auto;
padding: 0.35rem 0.7rem;
border: 1px solid var(--line);
border-radius: var(--radius-sm);
background: transparent;
color: var(--ink-soft);
font: inherit;
font-weight: 600;
font-size: 0.82rem;
white-space: nowrap;
cursor: pointer;
}
.git-source-remove svg { width: 14px; height: 14px; display: block; }
.git-source-remove:hover:not(:disabled) { background: var(--err-bg); color: var(--err-ink); border-color: var(--err-line); }
.git-source-remove:disabled { opacity: 0.5; cursor: wait; }
/* Per-row delete failure (role=alert): the err pair, inline after the
(re-enabled) button. */
.git-source-row-error {
margin-left: 0.6rem;
background: var(--err-bg);
color: var(--err-ink);
border: 1px solid var(--err-line);
border-radius: var(--radius-sm);
padding: 0.3rem 0.6rem;
font-size: 0.8rem;
font-weight: 600;
}
/* Env-fallback rows carry no Remove (nothing is stored to remove) —
the tag says where the row comes from (brand pair, 6.9:1). */
.git-source-env-tag {
color: var(--brand-ink);
background: var(--brand-soft);
border-radius: 999px;
padding: 0.2rem 0.6rem;
font-size: 0.78rem;
font-weight: 600;
}
/* Empty state: the tuning page's quiet centered line at full table
width — no stored rows AND no env fallback to show. */
.git-sources-empty {
margin: 0;
padding: 1.4rem 1rem;
text-align: center;
color: var(--ink-soft);
font-style: italic;
background: var(--surface);
border: 1px solid var(--line);
border-radius: var(--radius);
box-shadow: var(--shadow);
}
/* ---------- Document viewer (phase 10; two-row header since phase 34) ---------- */
/* Phase 34 (owner confirmation 2026-08-26): the viewer header is TWO
rows in one sticky <header> — row 1 reuses the standard .app-header /
@@ -1660,10 +1907,14 @@ details.thinking .thinking-text ul { margin: 0 0 0.5rem; }
— and let the brand wordmark (base ellipsis) absorb any remainder.
The ≤640 block below stays tighter and wins at phone widths. */
@media (max-width: 900px) {
.header-inner { gap: 0.65rem; }
.nav-link { padding: 0.4rem 0.6rem; font-size: 0.9rem; }
.app-nav { gap: 0.2rem; }
.new-chat-btn, .auth-link, .sync-btn, .steering-toggle { padding: 0.45rem 0.6rem; }
.header-inner { gap: 0.6rem; }
/* Phase 35: the admin-only "Git sources" link joins the nav — a
fourth text pill on the admin bar — so the tablet squeeze tightens
once more to hold 768px in the admin state (brand already the
designated clip target, pills squeeze next). */
.nav-link { padding: 0.4rem 0.5rem; font-size: 0.85rem; }
.app-nav { gap: 0.15rem; }
.new-chat-btn, .auth-link, .sync-btn, .steering-toggle { padding: 0.45rem 0.5rem; }
}
/* ---------- Responsive (mobile-first adjustments) ---------- */
@@ -1687,7 +1938,12 @@ details.thinking .thinking-text ul { margin: 0 0 0.5rem; }
18px brand mark) to hold 375px with the brand mark intact and
360px with the brand clipped clean (overflow:hidden — the mark
never overlaps the nav). */
.header-inner { gap: 0.3rem; }
/* Phase 35: the admin-only "Git sources" link joins the nav — a
fourth text pill on the admin bar — so the mobile squeeze tightens
once more (0.25rem inner gap, 0.72rem nav pills, 0.3rem pill
padding, 0.05rem nav gap) to hold 375px — and 360px, the brand
fully clipped — in the admin state without horizontal overflow. */
.header-inner { gap: 0.25rem; }
.brand { min-width: 0; overflow: hidden; }
.brand-mark { width: 18px; height: 18px; }
.brand-text {
@@ -1698,20 +1954,20 @@ details.thinking .thinking-text ul { margin: 0 0 0.5rem; }
text-overflow: ellipsis;
white-space: nowrap;
}
.nav-link { padding: 0.35rem 0.3rem; font-size: 0.78rem; }
.app-nav { gap: 0.1rem; }
.new-chat-btn { padding: 0.4rem 0.35rem; }
.nav-link { padding: 0.3rem 0.25rem; font-size: 0.72rem; }
.app-nav { gap: 0.05rem; }
.new-chat-btn { padding: 0.4rem 0.3rem; }
.new-chat-label { display: none; }
.new-chat-btn svg { display: block; }
/* Phase 16: the auth pill goes icon-only like New chat — brand text
ellipsizes as the designated squeeze target, no bar overflow. */
.auth-link { padding: 0.4rem 0.35rem; }
.auth-link { padding: 0.4rem 0.3rem; }
.auth-label { display: none; }
.auth-link svg { display: block; }
/* Phase 32: the sync pill goes icon-only like the other pills (the
aria-label keeps the accessible name); the spinning icon is the
visible running state on a touch screen. */
.sync-btn { padding: 0.4rem 0.35rem; }
.sync-btn { padding: 0.4rem 0.3rem; }
.sync-label { display: none; }
/* The last-result counts stay ANNOUNCED (aria-live is untouched) but
go visually hidden — the 58px bar has no room for the text; the
@@ -1725,7 +1981,7 @@ details.thinking .thinking-text ul { margin: 0 0 0.5rem; }
white-space: nowrap;
border: 0;
}
.steering-toggle { padding: 0.4rem 0.35rem; }
.steering-toggle { padding: 0.4rem 0.3rem; }
/* Visually hidden, NOT display:none — the accessible name keeps the
word "Tuning" next to the count badge. */
.steering-label {
@@ -1767,6 +2023,13 @@ details.thinking .thinking-text ul { margin: 0 0 0.5rem; }
.doc-modal-meta { padding-inline: 0.9rem; }
.doc-modal-content { padding: 0.75rem 0.9rem 1.25rem; }
.composer { padding: 0.5rem; }
/* Phase 35: the git sources add form stacks like the other cards —
label, full-width mono input, full-width button; the table
wrapper's horizontal scroll already covers long URLs. */
#git-source-form { flex-direction: column; align-items: stretch; }
#git-source-form > label { white-space: normal; }
#git-source-url { min-width: 0; }
#git-source-add { width: 100%; }
.footer-inner { flex-direction: column; gap: 0.2rem; text-align: center; }
main { padding-bottom: env(safe-area-inset-bottom, 0); }
}
frontend/document.html
+5
View File
@@ -35,6 +35,11 @@
reveals it once whoami says admin. The soft-gated page
itself is unchanged. -->
<a href="/sources.html" class="nav-link" id="nav-sources" hidden>Sources</a>
<!-- Phase 35 (owner permission 2026-08-26): the Git sources
link is admin-only — hidden by default, header.js
reveals it once whoami says admin, exactly like the
Sources link above. -->
<a href="/git-sources.html" class="nav-link" id="nav-git-sources" hidden>Git sources</a>
<!-- Phase 29 (now every page — phase 34, owner confirmation
2026-08-26): the Global Tuning link is admin-only (owner
permission 2026-08-25) — hidden by default, header.js
frontend/git-sources.html
+229
View File
@@ -0,0 +1,229 @@
<!doctype html>
<html lang="en">
<head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1, viewport-fit=cover">
<meta name="description" content="Add and remove the git repositories Brain of Reese syncs and indexes (admin-only).">
<title>Git sources · Brain of Reese</title>
<link rel="icon" href="data:image/svg+xml,%3Csvg%20xmlns=%22http://www.w3.org/2000/svg%22%20viewBox=%220%200%2064%2064%22%3E%3Cpath%20d=%22M32%204%2055%2018v28L32%2060%209%2046V18Z%22%20fill=%22%23121a2e%22%20stroke=%22%236d78f2%22%20stroke-width=%224%22%20stroke-linejoin=%22round%22/%3E%3Ccircle%20cx=%2232%22%20cy=%2232%22%20r=%226.5%22%20fill=%22%236d78f2%22/%3E%3Cpath%20d=%22M32%2025.5V16M32%2048v-9.5M25.5%2032H16M48%2032h-9.5%22%20stroke=%22%2322d3ee%22%20stroke-width=%223%22%20stroke-linecap=%22round%22/%3E%3C/svg%3E">
<link rel="stylesheet" href="/assets/styles.css">
</head>
<body>
<a class="skip-link" href="#main">Skip to content</a>
<!-- Phase 35: the SAME full header block every other page ships
(phase 34, owner confirmation 2026-08-26) — one shared owner of
the controls (assets/header.js via git-sources.js's relative
import). The admin-only "Git sources" nav link (#nav-git-sources)
joins this nav in phase 35 task 05, so it is NOT in this file
yet — the page lands without it, exactly like the other pages
land without the links task 05 adds to them. -->
<header class="app-header">
<div class="container header-inner">
<span class="brand">
<svg class="brand-mark" aria-hidden="true" viewBox="0 0 64 64"><path d="M32 4 55 18v28L32 60 9 46V18Z" fill="#121a2e" stroke="#6d78f2" stroke-width="4" stroke-linejoin="round"/><circle cx="32" cy="32" r="6.5" fill="#6d78f2"/><path d="M32 25.5V16M32 48v-9.5M25.5 32H16M48 32h-9.5" stroke="#22d3ee" stroke-width="3" stroke-linecap="round"/></svg>
<span class="brand-text">Brain of <strong>Reese</strong></span>
</span>
<nav class="app-nav" aria-label="Primary">
<a href="/" class="nav-link">Chat</a>
<!-- Phase 19 (now every page — phase 34, owner confirmation
2026-08-26): the Sources link is admin-only (owner
permission 2026-08-23) — hidden by default, header.js
reveals it once whoami says admin. -->
<a href="/sources.html" class="nav-link" id="nav-sources" hidden>Sources</a>
<!-- Phase 35 (owner permission 2026-08-26): the Git sources
link is admin-only — hidden by default, header.js
reveals it once whoami says admin, exactly like the
Sources link above; this page IS the current one, so the
link carries is-active + aria-current like Tuning on
tuning.html. -->
<a href="/git-sources.html" class="nav-link is-active" aria-current="page" id="nav-git-sources" hidden>Git sources</a>
<!-- Phase 29 (now every page — phase 34, owner confirmation
2026-08-26): the Global Tuning link is admin-only (owner
permission 2026-08-25) — hidden by default, header.js
reveals it once whoami says admin. -->
<a href="/tuning.html" class="nav-link" id="nav-tuning" hidden>Tuning</a>
</nav>
<!-- Phase 15 (now every page — phase 34, owner confirmation
2026-08-26): open the tuning-notes panel (stored in
Postgres, read into every system prompt). The behavior is
owned by the shared header module (assets/header.js); the
#steering-panel section ships in every page's <main>. -->
<button type="button" class="steering-toggle" id="steering-toggle"
aria-expanded="false" aria-controls="steering-panel">
<svg aria-hidden="true" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="1.7" stroke-linecap="round"><path d="M4 7h10M18 7h2M4 17h4M12 17h8"/><circle cx="15.5" cy="7" r="2.2"/><circle cx="9.5" cy="17" r="2.2"/></svg>
<span class="steering-label">Tuning</span>
<span class="steering-count" id="steering-count">0</span>
</button>
<!-- Phase 32 (now every page — phase 34, owner confirmation
2026-08-26): the admin-only "Sync sources" button — SHIPS
hidden (anonymous-safe), header.js reveals it for the admin
on the SAME cached whoami. The §7.4 "never stale" lifecycle
is module-owned (assets/header.js). This page's hint box
points at it: it is the action that clones the listed repos
and prunes the removed ones. -->
<button type="button" class="sync-btn" id="sync-btn" hidden aria-label="Sync sources">
<svg class="sync-icon" aria-hidden="true" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="1.7" stroke-linecap="round" stroke-linejoin="round"><path d="M21 12a9 9 0 1 1-9-9c2.52 0 4.93 1 6.74 2.74L21 8"/><path d="M21 3v5h-5"/></svg>
<span class="sync-label" id="sync-label">Sync sources</span>
</button>
<!-- Phase 14 (now every page — phase 34, owner confirmation
2026-08-26; module-owned since phase 34 task 02): on a
non-chat page "New chat" means "go to the chat, fresh" (the
module clears the key + navigates). -->
<button type="button" class="new-chat-btn" id="new-chat-btn" aria-label="New chat">
<svg aria-hidden="true" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="1.7" stroke-linecap="round"><path d="M12 5v14M5 12h14"/></svg>
<span class="new-chat-label">New chat</span>
</button>
<!-- Phase 16: single-admin auth — exactly one of Sign in / Sign
out is visible; /api/whoami decides at load (the shared
header module). Icon-only below 640px (aria-labels keep the
accessible names). -->
<a href="/login.html?next=/git-sources.html" class="auth-link" id="sign-in-link" hidden>
<svg aria-hidden="true" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="1.7" stroke-linecap="round" stroke-linejoin="round"><path d="M10 4h8a2 2 0 0 1 2 2v12a2 2 0 0 1-2 2h-8"/><path d="M4 12h11"/><path d="m12 9 3 3-3 3"/></svg>
<span class="auth-label">Sign in</span>
</a>
<button type="button" class="auth-link" id="sign-out-btn" aria-label="Sign out" hidden>
<svg aria-hidden="true" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="1.7" stroke-linecap="round" stroke-linejoin="round"><path d="M14 4H6a2 2 0 0 0-2 2v12a2 2 0 0 0 2 2h8"/><path d="M9 12h11"/><path d="m17 9 3 3-3 3"/></svg>
<span class="auth-label">Sign out</span>
</button>
</div>
</header>
<main id="main" class="app-main" tabindex="-1">
<!-- Phase 15 (now every page — phase 34, owner confirmation
2026-08-26): the tuning-notes panel — first child of <main>
on the non-chat pages, rendered + driven by assets/header.js
(shared), not the page script. -->
<section class="steering-panel" id="steering-panel" role="region"
aria-label="Tuning notes" hidden>
<div class="steering-panel-head">
<h2 class="steering-panel-title">Tuning notes</h2>
<p class="steering-panel-sub">Every note below steers all future answers.</p>
</div>
<ul class="steering-list" id="steering-list"></ul>
<p class="steering-empty" id="steering-empty">No tuning notes yet — press “Tune” under any answer to add one.</p>
</section>
<p class="visually-hidden" id="steering-announcer" role="status" aria-live="polite" aria-atomic="true"></p>
<div class="container git-sources-shell">
<!-- Phase 35: anonymous sign-in gate — the EXACT #sources-gate
pattern (phase 16) and the same .sources-gate visual
language: the page is the same shape as Sources. Visible
for anonymous, hidden for the admin (git-sources.js). The
catalog of git sources is what the login locks — chat stays
open to everyone (the soft rule). -->
<section class="sources-gate" id="git-sources-gate" aria-labelledby="git-sources-gate-title" hidden>
<div class="sources-gate-glyph" aria-hidden="true">
<svg viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="1.6" stroke-linecap="round" stroke-linejoin="round"><rect x="4" y="10" width="16" height="10" rx="2"/><path d="M8 10V7a4 4 0 0 1 8 0v3"/><circle cx="12" cy="14.5" r="1.4" fill="currentColor" stroke="none"/><path d="M12 16v2"/></svg>
</div>
<h2 id="git-sources-gate-title">Sign in to manage the git sources</h2>
<p class="sources-gate-sub">
The list of repositories the <strong>Sync sources</strong> button
clones and indexes is admin-only. Chat — and any document an
answer cites — stays open to everyone.
</p>
<a class="sources-gate-link" href="/login.html?next=/git-sources.html">Sign in</a>
</section>
<!-- Phase 35: the manager — SHIPS hidden (anonymous-safe; the
gate is what anonymous visitors see). git-sources.js
reveals it once the cached whoami says admin, then loads
the list. Full-width table on the 72rem frame — the
Sources-page pattern, no skinny single-column list. -->
<div id="git-sources-content" hidden>
<div class="page-head">
<h1>Git sources</h1>
<p class="page-sub">
The repositories the Sync button clones and indexes. Add or
remove them here — no <code>.env</code>, no restart.
</p>
</div>
<!-- Load failure (role=alert) with a retry — a GET /api/git-sources
non-2xx or network failure must never leave a stuck page.
git-sources.js fills #git-sources-load-error-text. -->
<div class="git-source-load-error" id="git-sources-load-error" role="alert" hidden>
<span id="git-sources-load-error-text"></span>
<button type="button" id="git-sources-retry">Try again</button>
</div>
<!-- Env-fallback note (phase locked decision): while the
git_sources table is EMPTY the list above comes from
BOR_GIT_SOURCES in .env (from_env: true) — the note says
so, and that adding or removing here switches management
to the database. Hidden by default; git-sources.js shows
it off the API's from_env flag. -->
<p class="git-source-env-note" id="git-sources-env-note" role="note" hidden>
These sources currently come from <code>BOR_GIT_SOURCES</code> in
<code>.env</code> — adding or removing one here switches management
to the database.
</p>
<!-- Add form: visible label + mono URL input + brand button
(dark ink on brand 5.2:1). §7.4 never-stale: the button
disables + relabels "Adding…" while the POST is in flight
and re-enables on success AND failure (the input is kept
on failure, same as the tuning forms). -->
<form id="git-source-form">
<label for="git-source-url">Add a git source</label>
<input
id="git-source-url"
name="url"
type="text"
maxlength="500"
autocomplete="off"
placeholder="https://github.com/you/homelab.git"
required
>
<button type="submit" id="git-source-add">Add source</button>
<p class="git-source-error" id="git-source-error" role="alert" hidden></p>
</form>
<div class="table-wrap" id="git-sources-table-wrap" role="region" aria-label="Git sources" tabindex="0">
<table class="git-sources-table" id="git-sources-table">
<caption class="visually-hidden">Git repositories the Sync button clones and indexes</caption>
<thead>
<tr>
<th scope="col">URL</th>
<th scope="col">Added</th>
<th scope="col">Actions</th>
</tr>
</thead>
<tbody id="git-sources-tbody"></tbody>
</table>
</div>
<!-- Empty state — no stored rows AND no env fallback. With
from_env, the env note above already explains where the
active list comes from. -->
<p class="git-sources-empty" id="git-sources-empty" hidden>No git sources stored yet.</p>
<!-- Scope boundary (phase locked decision): adding/removing a
repo does NOT clone or prune — the Sync button performs
that. The hint says so. -->
<p class="git-source-hint" id="git-sources-hint" role="note">
Use the <strong>Sync sources</strong> button in the header (or on
the Sources page) to clone the repos and refresh the index —
removing a repository prunes its documents from the index on the
next sync.
</p>
</div>
<!-- Polite live region: the screen-reader confirmation for list
loads, adds, and removals (git-sources.js owns the text). -->
<p class="visually-hidden" id="git-sources-announcer" role="status" aria-live="polite"></p>
</div>
</main>
<footer class="app-footer">
<div class="container footer-inner">
<span>Powered by Reese's self-hosted models</span>
</div>
</footer>
<!-- Phase 35: the page module loads the shared header through its
own `import "./header.js"` — a hoisted import evaluated before
this body runs (the single-evaluation design: no direct
header.js <script> tag; esbuild inlines it into the page
bundle in the image build). -->
<script type="module" src="/assets/git-sources.js"></script>
</body>
</html>
frontend/index.html
+5
View File
@@ -25,6 +25,11 @@
reveals it once whoami says admin. The soft-gated page
itself is unchanged. -->
<a href="/sources.html" class="nav-link" id="nav-sources" hidden>Sources</a>
<!-- Phase 35 (owner permission 2026-08-26): the Git sources
link is admin-only — hidden by default, header.js
reveals it once whoami says admin, exactly like the
Sources link above. -->
<a href="/git-sources.html" class="nav-link" id="nav-git-sources" hidden>Git sources</a>
<!-- Phase 29 (now every page — phase 34, owner confirmation
2026-08-26): the Global Tuning link is admin-only (owner
permission 2026-08-25) — hidden by default, header.js
frontend/login.html
+7
View File
@@ -26,6 +26,13 @@
reveals it once whoami says admin. The soft-gated page
itself is unchanged. -->
<a href="/sources.html" class="nav-link" id="nav-sources" hidden>Sources</a>
<!-- Phase 35 (owner permission 2026-08-26): the Git sources
link is admin-only — hidden by default, header.js
reveals it once whoami says admin, exactly like the
Sources link above. The phase-34 identical-header contract
requires it here too: every page's nav carries the same
four links (Chat, Sources, Git sources, Tuning). -->
<a href="/git-sources.html" class="nav-link" id="nav-git-sources" hidden>Git sources</a>
<!-- Phase 29 (now every page — phase 34, owner confirmation
2026-08-26): the Global Tuning link is admin-only (owner
permission 2026-08-25) — hidden by default, header.js
frontend/sources.html
+5
View File
@@ -25,6 +25,11 @@
reveals it once whoami says admin. The soft-gated page
itself is unchanged. -->
<a href="/sources.html" class="nav-link is-active" aria-current="page" id="nav-sources" hidden>Sources</a>
<!-- Phase 35 (owner permission 2026-08-26): the Git sources
link is admin-only — hidden by default, header.js
reveals it once whoami says admin, exactly like the
Sources link above. -->
<a href="/git-sources.html" class="nav-link" id="nav-git-sources" hidden>Git sources</a>
<!-- Phase 29 (now every page — phase 34, owner confirmation
2026-08-26): the Global Tuning link is admin-only (owner
permission 2026-08-25) — hidden by default, header.js
frontend/tuning.html
+5
View File
@@ -25,6 +25,11 @@
reveals it once whoami says admin. The soft-gated page
itself is unchanged. -->
<a href="/sources.html" class="nav-link" id="nav-sources" hidden>Sources</a>
<!-- Phase 35 (owner permission 2026-08-26): the Git sources
link is admin-only — hidden by default, header.js
reveals it once whoami says admin, exactly like the
Sources link above. -->
<a href="/git-sources.html" class="nav-link" id="nav-git-sources" hidden>Git sources</a>
<!-- Phase 29 (now every page — phase 34, owner confirmation
2026-08-26): the Global Tuning link is admin-only (owner
permission 2026-08-25) — hidden by default, header.js
scripts/import_docs.py
+24 -11
View File
@@ -7,15 +7,18 @@ Examples::
uv run python -m scripts.import_docs --prune # also drop deleted/out-of-scope files
uv run python -m scripts.import_docs --limit 5 # debug: first 5 files only
Source resolution (phase 28), in precedence order:
Source resolution (phase 28, extended in phase 35), in precedence order:
1. ``--source PATH`` — explicit manual directories (repeatable) always win;
``BOR_GIT_SOURCES`` is ignored when this flag is used.
2. ``BOR_GIT_SOURCES`` (comma-separated git URLs) — each repo is cloned
(first run, shallow ``--depth 1``) or fast-forwarded (``git pull
--ff-only``) into ``BOR_SOURCES_DIR/<repo-name>/`` (default
``~/bor-sources``) and the resulting checkouts are imported. A failing
clone/pull aborts the whole run *before* anything is imported.
git sources are ignored when this flag is used.
2. The effective git sources — the admin-managed ``git_sources`` table
rows, else the ``BOR_GIT_SOURCES`` (comma-separated) fallback
(:func:`app.rag.git_sources.effective_git_sources`, the same shared
resolver the in-app Sync button uses) — each repo is cloned (first
run, shallow ``--depth 1``) or fast-forwarded (``git pull --ff-only``)
into ``BOR_SOURCES_DIR/<repo-name>/`` (default ``~/bor-sources``) and
the resulting checkouts are imported. A failing clone/pull aborts the
whole run *before* anything is imported.
3. Fallback — the legacy ``DEFAULT_SOURCES`` (``~/Homelab`` +
``~/Deployments``), kept for backwards compatibility.
@@ -53,6 +56,7 @@ from app.core.debugging import configure_debugging
from app.core.logging import configure_logging
from app.db import SessionLocal
from app.models import KbOverview
from app.rag.git_sources import effective_git_sources
from app.rag.importer import ImportSummary, import_sources
from app.rag.llm import LLMClient
from app.rag.overview import regenerate_overview
@@ -111,20 +115,29 @@ def repo_name(url: str) -> str:
def _resolve_sources(cli_sources: list[Path] | None, settings: Settings) -> list[Path]:
"""Resolve the directories to import (phase 28).
"""Resolve the directories to import (phase 28, extended in phase 35).
Precedence: ``--source`` (explicit manual paths — always wins) >
``BOR_GIT_SOURCES`` (each URL cloned/pulled via
the effective git sources — the ``git_sources`` DB rows, else the
``BOR_GIT_SOURCES`` fallback
(:func:`app.rag.git_sources.effective_git_sources`; the import needs
the database anyway, so resolution opens a short session and there
is no DB-down branch) — each URL cloned/pulled via
:func:`scripts.git_sync.clone_or_pull` into
``BOR_SOURCES_DIR/<repo-name>/``) > the legacy ``DEFAULT_SOURCES``.
``BOR_SOURCES_DIR/<repo-name>/`` > the legacy ``DEFAULT_SOURCES``.
A :class:`GitSyncError` from a failing clone/pull propagates to
:func:`main`, which aborts the run before importing anything.
"""
if cli_sources:
return [path.expanduser() for path in cli_sources]
git_urls = settings.git_source_list
db = SessionLocal()
try:
git_urls, origin = effective_git_sources(db)
finally:
db.close()
if git_urls:
logger.info("git sources: %d repo(s) origin=%s", len(git_urls), origin)
sources_root = Path(settings.sources_dir).expanduser()
return [clone_or_pull(url, sources_root / repo_name(url)) for url in git_urls]
return [path.expanduser() for path in DEFAULT_SOURCES]
tests/e2e/test_git_sources_admin.py
+572
View File
@@ -0,0 +1,572 @@
"""Phase 35 story E2E (Playwright): the admin page to add / remove git
sources (``/git-sources.html``).
Story: ``.agent/user_stories/git-sources-admin.md``
Run in isolation (DB must be up: ``podman compose up -d db``):
uv run pytest tests/e2e/test_git_sources_admin.py -v --no-cov
The story gate for the admin-managed git source list. **No git, no
network** — this suite is UI + API only (the clone/import pipeline is
mocked at the integration level, phase 35 task 03; a real sync is
deliberately never triggered here). The stored list (``git_sources``
table, migration 0006) is exercised through the real page and the real
admin-only CRUD API (phase 35 task 02).
Per-module app env (the conftest pattern, module-scoped — as in
``test_sync_button.py``): this story's app boots with a fixed
``BOR_GIT_SOURCES`` CSV (two deterministic URLs that are NEVER cloned —
nothing in this suite triggers a sync) so the empty-table env fallback
(the phase's locked decision: ``BOR_GIT_SOURCES`` only applies while the
``git_sources`` table is empty) can be asserted against real env rows.
The session app (no git sources) is never started in this isolated run,
so no port clash.
Contract under test:
* anonymous: the sign-in gate (the exact ``#sources-gate`` pattern), the
manager hidden (list + add form inert), NO ``/api/git-sources`` call,
403 on all three routes (asserted with the page context's request
client, the ``test_admin_auth.py``/``test_sync_button.py`` pattern),
and ``#nav-git-sources`` hidden on all five pages;
* admin: ``#nav-git-sources`` visible on all five pages (revealed by the
shared header on the cached whoami), clicking it from the chat lands
on ``/git-sources.html`` with the link ``is-active``; the stored rows
render as a full-width table (mono URL, added date, per-row Remove)
with the env note hidden while the DB has rows; add (201 → row, input
cleared, button re-enabled), duplicate (inline role=alert, no new
row), invalid shape (inline 422, no new row), remove (confirm → gone;
cancel → stays); with the table truncated the env rows render with
"from .env" tags and ``#git-sources-env-note`` visible;
* the page a11y / no-CDN basics (UI Structure Check, AGENTS.md rule 5):
landmarks, labeled form control, full-width table, ≥44px targets,
3px focus-visible outline, the aria-live list announcer, same-origin
assets only (rule 6).
Test → story mapping (Playwright Mapping Rule):
1. ``test_anonymous_gate_no_api_calls_and_403s``
2. ``test_admin_nav_link_on_all_five_pages_and_click_navigates``
3. ``test_admin_sees_seeded_rows_with_dates_and_no_env_note``
4. ``test_admin_add_then_remove_lifecycle``
5. ``test_admin_env_fallback_rows_and_note``
6. ``test_admin_page_a11y_and_no_cdn``
"""
from __future__ import annotations
import os
import re
import subprocess
import sys
import uuid
from collections.abc import Iterator
from datetime import UTC, datetime, timedelta
from pathlib import Path
from typing import Any
import pytest
from playwright.sync_api import Dialog, Page, expect
from sqlalchemy import text
from app.db import SessionLocal
from app.models import GitSource
from e2e.auth_helpers import login
from e2e.conftest import (
ADMIN_PASSWORD,
APP_PORT,
SESSION_SECRET,
USE_REAL_LLM,
_wait_http,
)
REPO = Path(__file__).resolve().parents[2]
APP_URL = f"http://127.0.0.1:{APP_PORT}"
#: The five pages that ship the header (phase 34 contract) — the pages
#: task 05 gave the admin-only "Git sources" nav link.
CHAT_URL = "/"
SOURCES_URL = "/sources.html"
VIEWER_URL = "/document.html?source=docs&path=homelab%2Fkubernetes.md"
TUNING_URL = "/tuning.html"
GIT_SOURCES_URL = "/git-sources.html"
FIVE_PAGES = (
("chat", CHAT_URL),
("sources", SOURCES_URL),
("viewer", VIEWER_URL),
("tuning", TUNING_URL),
("git-sources", GIT_SOURCES_URL),
)
#: The module app's ``BOR_GIT_SOURCES`` — two deterministic URLs that
#: are NEVER cloned (no sync is triggered in this suite; the fallback
#: list is what is under test).
ENV_SOURCE_A = "https://github.com/reese/env-alpha.git"
ENV_SOURCE_B = "https://github.com/reese/env-beta.git"
ENV_SOURCES_CSV = ",".join((ENV_SOURCE_A, ENV_SOURCE_B))
#: ``is-active`` as a word-boundary regex — to_have_class() matches the
#: WHOLE class string, so the current-page marker is asserted the same
#: way test_tuning_nav_link.py does it.
IS_ACTIVE = re.compile(r"\bis-active\b")
#: Deterministic URLs for the UI-driven assertions.
SEED_URL = "https://gitlab.example.com/reese/seeded.git"
NEW_REPO_URL = "https://example.com/reese/new-repo.git"
# ---------------------------------------------------------------------------
# Fixtures
# ---------------------------------------------------------------------------
@pytest.fixture(scope="module")
def app_server(mock_llm: int) -> Iterator[str]:
"""The real app under test — per-module env: a fixed two-URL
``BOR_GIT_SOURCES`` CSV so the empty-table env fallback (the phase's
locked decision) is live. No ``BOR_SOURCES_DIR``, no git: nothing in
this suite clones or syncs."""
env = dict(os.environ)
env.pop("DEBUGPY", None)
env["BOR_ENVIRONMENT"] = "e2e"
env["BOR_STATIC_DIR"] = str(REPO / "frontend")
env["BOR_LLM_BASE_URL"] = (
"https://aipi.reeseapps.com/v1"
if USE_REAL_LLM
else f"http://127.0.0.1:{mock_llm}/v1"
)
# Mock-calibrated threshold (conftest pattern) — no chat turn is
# ever sent in this suite, but the app boots with the same env shape.
env["BOR_RELEVANCE_THRESHOLD"] = "0.30"
env.setdefault(
"BOR_DATABASE_URL",
"postgresql+psycopg://reese:reese@localhost:5432/brain_of_reese",
)
# Phase 16: admin auth must be set or create_app() refuses to boot.
env["BOR_ADMIN_PASSWORD"] = ADMIN_PASSWORD
env["BOR_SESSION_SECRET"] = SESSION_SECRET
# Phase 35: the empty-table fallback list (never cloned here).
env["BOR_GIT_SOURCES"] = ENV_SOURCES_CSV
proc = subprocess.Popen(
[sys.executable, "-m", "uvicorn", "app.main:app",
"--host", "127.0.0.1", "--port", str(APP_PORT), "--log-level", "warning"],
cwd=REPO,
env=env,
)
try:
_wait_http(f"{APP_URL}/api/health")
yield APP_URL
finally:
proc.terminate()
try:
proc.wait(timeout=10)
except subprocess.TimeoutExpired:
proc.kill()
@pytest.fixture(scope="module")
def app_url(app_server: str) -> str:
return app_server
def _truncate_git_sources() -> None:
with SessionLocal() as db:
db.execute(text("TRUNCATE git_sources"))
db.commit()
@pytest.fixture(autouse=True)
def _clean_git_sources(db_ready: None) -> Iterator[None]:
"""Fresh ``git_sources`` table per test — this suite owns the table
(the E2E isolation pattern); the KB tables are irrelevant here and
are left untouched (the viewer page may show its not-found state —
no document assertion is made about it). The table is emptied BOTH
before and after every test: suites run in isolation but share one
Postgres, and a leftover row would flip another suite's app from the
``BOR_GIT_SOURCES`` env fallback to the DB list (the sync story's
``effective_git_sources`` resolver reads the DB first)."""
_truncate_git_sources()
yield
_truncate_git_sources()
# ---------------------------------------------------------------------------
# Helpers
# ---------------------------------------------------------------------------
def _seed_rows(urls: list[str]) -> None:
"""Store rows directly (deterministic, ordered added_at) — the task
allows seeding via the API *or* SessionLocal; direct inserts keep
the list-rendering test independent of the add/remove lifecycle
test."""
base = datetime(2026, 8, 26, 9, 0, 0, tzinfo=UTC)
with SessionLocal() as db:
for i, url in enumerate(urls):
db.add(GitSource(url=url, added_at=base + timedelta(minutes=i)))
db.commit()
def _admin_git_sources_page(page: Page, app_url: str) -> None:
"""Real form login landing on the git sources page (admin settled:
Sign out visible, the manager revealed by the page module)."""
login(page, app_url, next=GIT_SOURCES_URL)
expect(page).to_have_url(app_url + GIT_SOURCES_URL, timeout=30_000)
expect(page.locator("#sign-out-btn")).to_be_visible(timeout=15_000)
expect(page.locator("#git-sources-gate")).to_be_hidden()
expect(page.locator("#git-sources-content")).to_be_visible()
def _row(page: Page, url: str) -> Any:
"""The table row whose mono URL cell shows ``url``."""
return page.locator("#git-sources-tbody tr", has_text=url)
def _assert_settled(page: Page, admin: bool) -> None:
"""The shared header's whoami toggle has landed (one of Sign in /
Sign out visible) — the same settled-state gate as
test_nav_consistency.py."""
if admin:
expect(page.locator("#sign-out-btn")).to_be_visible(timeout=15_000)
expect(page.locator("#sign-in-link")).to_be_hidden()
else:
expect(page.locator("#sign-in-link")).to_be_visible(timeout=15_000)
expect(page.locator("#sign-out-btn")).to_be_hidden()
# ---------------------------------------------------------------------------
# 1. Anonymous: gate, inert manager, no API calls, 403s, hidden link
# ---------------------------------------------------------------------------
def test_anonymous_gate_no_api_calls_and_403s(
page: Page, app_url: str, db_ready: None
) -> None:
page.set_default_timeout(30_000)
# Track every /api/git-sources request the page itself makes — the
# gate must be reached WITHOUT touching the admin API (the
# test_admin_auth.py pattern for the Sources page's /api/docs).
api_calls: list[str] = []
page.on(
"request",
lambda r: api_calls.append(r.url) if "/api/git-sources" in r.url else None,
)
page.goto(app_url + GIT_SOURCES_URL)
_assert_settled(page, admin=False)
# The sign-in gate (the #sources-gate pattern, phase 16)…
gate = page.locator("#git-sources-gate")
expect(gate).to_be_visible()
expect(gate).to_contain_text("Sign in to manage the git sources")
link = gate.locator("a[href='/login.html?next=/git-sources.html']")
expect(link).to_have_count(1)
box = link.bounding_box()
assert box is not None and box["height"] >= 44, f"gate link too small: {box}"
# …and the manager is absent/inert: list, add form, env note — all
# inside the hidden #git-sources-content.
expect(page.locator("#git-sources-content")).to_be_hidden()
expect(page.locator("#git-sources-table")).to_be_hidden()
expect(page.locator("#git-source-form")).to_be_hidden()
expect(page.locator("#git-sources-env-note")).to_be_hidden()
# The admin-only nav link is hidden on this page…
expect(page.locator("#nav-git-sources")).to_be_hidden()
# …and on the other four pages (all five ship it hidden by default).
for _name, path in FIVE_PAGES:
if path == GIT_SOURCES_URL:
continue
page.goto(app_url + path)
_assert_settled(page, admin=False)
expect(page.locator("#nav-git-sources")).to_be_hidden()
# The gate never called the admin API…
assert api_calls == [], f"anonymous page called the git sources API: {api_calls}"
# …and the API 403s anonymous callers on all three routes (the
# test_sync_button.py pattern — the page context has no cookie).
assert page.request.get(f"{app_url}/api/git-sources").status == 403
assert (
page.request.post(
f"{app_url}/api/git-sources", data={"url": NEW_REPO_URL}
).status
== 403
)
assert page.request.delete(f"{app_url}/api/git-sources/{uuid.uuid4()}").status == 403
# ---------------------------------------------------------------------------
# 2. Admin: the nav link on all five pages; the click lands + is-active
# ---------------------------------------------------------------------------
def test_admin_nav_link_on_all_five_pages_and_click_navigates(
page: Page, app_url: str, db_ready: None
) -> None:
page.set_default_timeout(30_000)
login(page, app_url, next=CHAT_URL)
expect(page).to_have_url(app_url + CHAT_URL, timeout=30_000)
# The link is revealed (admin) on every one of the five pages,
# pointing at the git sources page, labeled "Git sources" — and it
# is NOT the current page on the four non-git-sources pages.
for _name, path in FIVE_PAGES:
if path != CHAT_URL:
page.goto(app_url + path)
_assert_settled(page, admin=True)
link = page.locator("#nav-git-sources")
expect(link).to_be_visible(timeout=15_000)
expect(link).to_have_attribute("href", GIT_SOURCES_URL)
expect(link).to_have_text("Git sources")
if path != GIT_SOURCES_URL:
expect(link).not_to_have_class(IS_ACTIVE)
# From the chat: a real click navigates…
page.goto(app_url + CHAT_URL)
_assert_settled(page, admin=True)
page.click("#nav-git-sources")
expect(page).to_have_url(app_url + GIT_SOURCES_URL, timeout=30_000)
# …where the page's own link is the current-page marker.
link = page.locator("#nav-git-sources")
expect(link).to_be_visible(timeout=15_000)
expect(link).to_have_class(IS_ACTIVE)
expect(link).to_have_attribute("aria-current", "page")
# ---------------------------------------------------------------------------
# 3. Admin: two seeded rows render (mono URL + added date); env note
# hidden while the DB has rows
# ---------------------------------------------------------------------------
def test_admin_sees_seeded_rows_with_dates_and_no_env_note(
page: Page, app_url: str, db_ready: None
) -> None:
page.set_default_timeout(30_000)
_seed_rows([SEED_URL, ENV_SOURCE_A]) # ordered by added_at
_admin_git_sources_page(page, app_url)
# Both rows render, oldest first…
expect(page.locator("#git-sources-tbody tr")).to_have_count(2)
first = _row(page, SEED_URL)
second = _row(page, ENV_SOURCE_A)
expect(first.locator("td.git-source-url-cell code")).to_have_text(SEED_URL)
expect(second.locator("td.git-source-url-cell code")).to_have_text(ENV_SOURCE_A)
expect(page.locator("#git-sources-tbody tr").first).to_contain_text(SEED_URL)
expect(page.locator("#git-sources-tbody tr").last).to_contain_text(ENV_SOURCE_A)
# …each with the mono URL cell, a rendered added date (not the "—"
# env-fallback placeholder), and a labeled per-row Remove button.
for row, url in ((first, SEED_URL), (second, ENV_SOURCE_A)):
added = row.locator("td").nth(1)
expect(added).not_to_be_empty()
expect(added).not_to_have_text("—")
remove = row.locator(".git-source-remove")
expect(remove).to_have_count(1)
expect(remove).to_have_attribute("aria-label", f"Remove git source: {url}")
# DB rows exist → from_env is false: the env note is hidden and the
# empty state too.
expect(page.locator("#git-sources-env-note")).to_be_hidden()
expect(page.locator("#git-sources-empty")).to_be_hidden()
# ---------------------------------------------------------------------------
# 4. Admin: add → duplicate → invalid → remove (accept + cancel) — the
# full never-stale lifecycle
# ---------------------------------------------------------------------------
def test_admin_add_then_remove_lifecycle(
page: Page, app_url: str, db_ready: None
) -> None:
page.set_default_timeout(30_000)
_seed_rows([SEED_URL])
_admin_git_sources_page(page, app_url)
expect(page.locator("#git-sources-tbody tr")).to_have_count(1)
error = page.locator("#git-source-error")
add_btn = page.locator("#git-source-add")
# --- add: 201 → the row appears, the input clears, the button
# re-enables (never stale) ---------------------------------------
page.fill("#git-source-url", NEW_REPO_URL)
add_btn.click()
expect(page.locator("#git-sources-tbody tr")).to_have_count(2, timeout=30_000)
expect(_row(page, NEW_REPO_URL).locator("td.git-source-url-cell code")).to_have_text(
NEW_REPO_URL
)
expect(page.locator("#git-source-url")).to_have_value("")
expect(add_btn).to_be_enabled()
expect(add_btn).to_have_text("Add source")
# A stored row exists → the env note stays hidden.
expect(page.locator("#git-sources-env-note")).to_be_hidden()
# --- duplicate: inline role=alert, NO new row, button re-enabled,
# the (failed) input survives for one edit ------------------------
page.fill("#git-source-url", NEW_REPO_URL)
add_btn.click()
expect(error).to_be_visible(timeout=30_000)
assert error.get_attribute("role") == "alert"
expect(error).to_contain_text("already exists")
expect(page.locator("#git-sources-tbody tr")).to_have_count(2)
expect(add_btn).to_be_enabled()
expect(add_btn).to_have_text("Add source")
expect(page.locator("#git-source-url")).to_have_value(NEW_REPO_URL)
# --- invalid shape: inline 422 (the server detail, never the
# echoed input), NO new row, button re-enabled --------------------
page.fill("#git-source-url", "not a valid url")
add_btn.click()
expect(error).to_be_visible(timeout=30_000)
expect(error).to_contain_text("not a valid git URL")
expect(page.locator("#git-sources-tbody tr")).to_have_count(2)
expect(add_btn).to_be_enabled()
expect(add_btn).to_have_text("Add source")
# --- remove: accept the confirm → the row disappears --------------
dialog_action: dict[str, bool] = {"accept": True}
deletes: list[str] = []
def handle_dialog(dialog: Dialog) -> None:
if dialog_action["accept"]:
dialog.accept()
else:
dialog.dismiss()
def track_delete(r: Any) -> None:
if r.method == "DELETE" and "/api/git-sources/" in r.url:
deletes.append(r.url)
page.on("dialog", handle_dialog)
page.on("request", track_delete)
try:
_row(page, NEW_REPO_URL).locator(".git-source-remove").click()
expect(page.locator("#git-sources-tbody tr")).to_have_count(1, timeout=30_000)
expect(_row(page, NEW_REPO_URL)).to_have_count(0)
# The seed row survived — and exactly one DELETE went out.
expect(_row(page, SEED_URL)).to_have_count(1)
assert len(deletes) == 1, f"expected one DELETE, saw: {deletes}"
# --- remove: cancel the confirm → the row stays, NO DELETE ----
dialog_action["accept"] = False
_row(page, SEED_URL).locator(".git-source-remove").click()
# Wait for the dialog round-trip to settle (dismiss → the JS
# returns without fetching) so the "no second DELETE" claim is
# made on a settled page.
page.wait_for_timeout(500)
assert len(deletes) == 1, f"canceled removal still deleted: {deletes}"
expect(_row(page, SEED_URL)).to_have_count(1)
expect(page.locator("#git-sources-tbody tr")).to_have_count(1)
finally:
page.remove_listener("dialog", handle_dialog)
page.remove_listener("request", track_delete)
# ---------------------------------------------------------------------------
# 5. Admin: empty table → the env rows render + the env note is visible
# (BOR_GIT_SOURCES is the empty-table fallback)
# ---------------------------------------------------------------------------
def test_admin_env_fallback_rows_and_note(
page: Page, app_url: str, db_ready: None
) -> None:
page.set_default_timeout(30_000)
# The autouse fixture truncated git_sources — the table is EMPTY, so
# the module app's BOR_GIT_SOURCES CSV is the effective list.
_admin_git_sources_page(page, app_url)
# The two env rows render (CSV order), each tagged "from .env" with
# no Remove (nothing is stored to remove) and the "—" added date.
expect(page.locator("#git-sources-tbody tr")).to_have_count(2)
expect(page.locator("#git-sources-tbody tr").first).to_contain_text(ENV_SOURCE_A)
expect(page.locator("#git-sources-tbody tr").last).to_contain_text(ENV_SOURCE_B)
for row in page.locator("#git-sources-tbody tr").all():
expect(row.locator(".git-source-env-tag")).to_have_count(1)
expect(row.locator(".git-source-env-tag")).to_have_text("from .env")
expect(row.locator(".git-source-remove")).to_have_count(0)
expect(row.locator("td").nth(1)).to_have_text("—")
# The env-fallback note explains the active list's origin…
expect(page.locator("#git-sources-env-note")).to_be_visible()
# …and the API agrees: from_env true, null ids, the env URLs.
r = page.request.get(f"{app_url}/api/git-sources")
assert r.status == 200, r.text
body = r.json()
assert body["from_env"] is True
assert [s["url"] for s in body["sources"]] == [ENV_SOURCE_A, ENV_SOURCE_B]
assert all(s["id"] is None and s["added_at"] is None for s in body["sources"])
# ---------------------------------------------------------------------------
# 6. UI Structure Check (AGENTS.md rule 5) + no CDN (rule 6)
# ---------------------------------------------------------------------------
def test_admin_page_a11y_and_no_cdn(
page: Page, app_url: str, db_ready: None
) -> None:
page.set_default_timeout(30_000)
_seed_rows([SEED_URL])
_admin_git_sources_page(page, app_url)
expect(page.locator("#git-sources-tbody tr")).to_have_count(1)
# Standard app frame: landmarks + skip link (PLAN §7.2).
expect(page.locator("header.app-header")).to_have_count(1)
expect(page.locator('nav[aria-label="Primary"]')).to_have_count(1)
expect(page.locator("main#main")).to_have_count(1)
expect(page.locator("footer.app-footer")).to_have_count(1)
expect(page.locator(".skip-link")).to_have_count(1)
# No CDN (rule 6): no https:// asset tags; every script/link ref is
# same-origin or a data: URI.
html = page.content()
assert 'src="https://' not in html and 'href="https://' not in html
refs = page.evaluate(
"""() => [...document.querySelectorAll("script[src], link[href]")]
.map((el) => el.src || el.href)"""
)
assert refs, "expected local asset references"
for ref in refs:
assert ref.startswith(app_url) or ref.startswith("data:"), (
f"non-local asset reference: {ref}"
)
# The form input is labeled (visible <label for=…>).
url_input = page.get_by_label("Add a git source")
expect(url_input).to_have_count(1)
# Full-width table (PLAN §7.1 — no skinny list): the table fills the
# 72rem container (≥80%, the Sources-page assertion).
table = page.locator("#git-sources-table")
expect(table).to_be_visible()
table_box = table.bounding_box()
shell_box = page.locator(".git-sources-shell").bounding_box()
assert table_box is not None and shell_box is not None
assert table_box["width"] >= 0.80 * shell_box["width"], (
f"table is {table_box['width']:.0f}px in a {shell_box['width']:.0f}px container"
)
# Touch targets ≥44px (add button + a row's Remove).
for el in (page.locator("#git-source-add"), page.locator(".git-source-remove")):
box = el.bounding_box()
assert box is not None and box["height"] >= 44, f"target too small: {box}"
# :focus-visible draws the 3px outline (the theme contract).
page.focus("#git-source-url")
outline = page.evaluate(
"() => getComputedStyle(document.querySelector('#git-source-url')).outlineWidth"
)
assert outline == "3px", f"focus-visible outline missing: {outline!r}"
# The list updates are announced (aria-live) and the error line is a
# role=alert region; the env note + hint are role=note.
announcer = page.locator("#git-sources-announcer")
assert announcer.get_attribute("role") == "status"
assert announcer.get_attribute("aria-live") == "polite"
assert page.locator("#git-source-error").get_attribute("role") == "alert"
assert page.locator("#git-sources-env-note").get_attribute("role") == "note"
assert page.locator("#git-sources-hint").get_attribute("role") == "note"
tests/e2e/test_nav_consistency.py
+16 -9
View File
@@ -18,15 +18,17 @@ sources, document viewer, global tuning, login): one shared markup block
Per role, the VISIBLE inventory:
* admin: brand + nav [Chat, #nav-sources, #nav-tuning] + #steering-toggle
+ #sync-btn + #new-chat-btn + #sign-out-btn (with #sign-in-link
hidden) — on all five pages, same id+class inventory, same DOM order;
* anonymous: brand + nav [Chat] (#nav-sources / #nav-tuning hidden —
locked A10 UI revision) + #new-chat-btn + #sign-in-link (with
#sync-btn hidden, #sign-out-btn hidden) on all five pages — and the
steering toggle + panel are ABSENT from the DOM (phase 16 "absent,
not hidden" treatment, carried into phase 34 task 01; test_admin_auth
pins it).
* admin: brand + nav [Chat, #nav-sources, #nav-git-sources, #nav-tuning]
(four links, that order — the Git sources link joined in phase 35,
owner permission 2026-08-26) + #steering-toggle + #sync-btn +
#new-chat-btn + #sign-out-btn (with #sign-in-link hidden) — on all
five pages, same id+class inventory, same DOM order;
* anonymous: brand + nav [Chat] (#nav-sources / #nav-git-sources /
#nav-tuning hidden — locked A10 UI revision) + #new-chat-btn +
#sign-in-link (with #sync-btn hidden, #sign-out-btn hidden) on all
five pages — and the steering toggle + panel are ABSENT from the DOM
(phase 16 "absent, not hidden" treatment, carried into phase 34 task
01; test_admin_auth pins it).
Normalization for the inventory comparison: the current-page ``is-active``
nav marker and the sign-in ``?next=`` value legitimately differ per page,
@@ -214,6 +216,9 @@ def _visit(page: Page, app_url: str, name: str, url: str, admin: bool) -> list[s
expect(page.locator(".app-nav a[href='/']")).to_be_visible() # Chat
if admin:
expect(page.locator("#nav-sources")).to_be_visible()
# Phase 35: the fourth admin-only nav link (Git sources) is
# revealed on every page, between Sources and Tuning.
expect(page.locator("#nav-git-sources")).to_be_visible()
expect(page.locator("#nav-tuning")).to_be_visible()
expect(page.locator("#steering-toggle")).to_be_visible()
expect(page.locator("#sync-btn")).to_be_visible()
@@ -223,6 +228,7 @@ def _visit(page: Page, app_url: str, name: str, url: str, admin: bool) -> list[s
# Locked A10 UI revision: admin-only links ship hidden, never
# revealed for anonymous…
expect(page.locator("#nav-sources")).to_be_hidden()
expect(page.locator("#nav-git-sources")).to_be_hidden()
expect(page.locator("#nav-tuning")).to_be_hidden()
expect(page.locator("#sync-btn")).to_be_hidden()
expect(page.locator("#sign-out-btn")).to_be_hidden()
@@ -281,6 +287,7 @@ def _admin_login_page_inventory(page: Page, app_url: str) -> list[str]:
_wait_settled(page, admin=True)
expect(page.locator(".app-nav a[href='/']")).to_be_visible()
expect(page.locator("#nav-sources")).to_be_visible()
expect(page.locator("#nav-git-sources")).to_be_visible()
expect(page.locator("#nav-tuning")).to_be_visible()
expect(page.locator("#steering-toggle")).to_be_visible()
expect(page.locator("#sync-btn")).to_be_visible()
tests/integration/test_api.py
+5 -1
View File
@@ -58,6 +58,7 @@ def test_suggestions_honors_bor_suggestions_env_override(monkeypatch) -> None:
("/document.html", "Brain of Reese"), # phase 10: viewer page
("/login.html", "Sign in"), # phase 16: admin sign-in page
("/tuning.html", "Global Tuning"), # phase 27: global tuning page
("/git-sources.html", "Git sources"), # phase 35: admin git sources page
],
)
def test_html_pages_served_locally_no_cdn(client, path: str, marker: str) -> None:
@@ -94,7 +95,7 @@ def test_index_page_no_cache_with_versioned_asset_refs(client) -> None:
@pytest.mark.parametrize(
"path",
["/sources.html", "/document.html", "/login.html", "/tuning.html"],
["/sources.html", "/document.html", "/login.html", "/tuning.html", "/git-sources.html"],
)
def test_html_pages_no_cache_with_versioned_refs(client, path: str) -> None:
"""Each of the other four pages revalidates and carries at least one
@@ -146,6 +147,7 @@ def test_styles_and_js_served(client) -> None:
assert client.get("/assets/login.js").status_code == 200 # phase 16: login page
assert client.get("/assets/document-modal.js").status_code == 200 # phase 26: modal module
assert client.get("/assets/tuning.js").status_code == 200 # phase 27: tuning page
assert client.get("/assets/git-sources.js").status_code == 200 # phase 35: git sources page
# Emoji code points banned from UI chrome (phase 08): the pictograph
@@ -177,12 +179,14 @@ def _find_emoji(text: str) -> list[str]:
"/document.html",
"/login.html", # phase 16
"/tuning.html", # phase 27
"/git-sources.html", # phase 35
"/assets/app.js",
"/assets/sources.js",
"/assets/markdown.js",
"/assets/document.js",
"/assets/login.js", # phase 16
"/assets/document-modal.js", # phase 26: the document modal module
"/assets/git-sources.js", # phase 35: the git sources page module
"/assets/styles.css",
],
)
tests/integration/test_git_sources_api.py
+302
View File
@@ -0,0 +1,302 @@
"""Integration: the admin git-sources CRUD API (phase 35, task 02).
Real Postgres (``podman compose up -d db``); the ``BOR_GIT_SOURCES``
fallback is exercised deterministically by monkeypatching the router's
``get_settings`` with a fresh ``Settings(_env_file=None, git_sources=…)``
(same pattern as ``test_sync_api.py`` — the dev ``.env`` never leaks in).
Contract under test:
* anonymous → 403 ``{"detail": "admin only"}`` on GET, POST, and DELETE
(phase 16 pattern, same as ``/api/sync``);
* GET — empty table + env set → the env rows with ``from_env: true`` and
null ``id``/``added_at``; empty table + empty env → ``sources: []``
with ``from_env: true``; any DB rows → ``from_env: false`` and the env
var is ignored (the phase's locked decision); DB rows ordered by
``(added_at, id)``;
* POST — 201 stored trimmed; duplicate (even with different surrounding
whitespace) → 409 with a generic detail that never echoes the URL
(credential safety), including when only the DB unique index catches
it; bad shape / blank / >500 chars → 422, also input-free;
* DELETE — 204 and gone; an emptied table falls back to the env list
again; unknown id → 404.
``git_sources`` is global state: truncated around every test.
"""
from __future__ import annotations
import uuid
from collections.abc import Iterator
from datetime import UTC, datetime, timedelta
from typing import Any
import pytest
from fastapi.testclient import TestClient
from sqlalchemy import select, text
from sqlalchemy.orm import Session
from app.api import git_sources as git_sources_api
from app.config import Settings
from app.models import GitSource
@pytest.fixture(autouse=True)
def clean_git_sources(db: Session) -> Iterator[None]:
"""The stored list is global state: reset around every test."""
db.execute(text("TRUNCATE git_sources"))
db.commit()
yield
db.execute(text("TRUNCATE git_sources"))
db.commit()
def _settings(git_sources: str = "") -> Settings:
"""Fresh settings with the ``.env`` file ignored; the explicit kwarg
beats any process env leaks (test_sync_api pattern)."""
return Settings(_env_file=None, git_sources=git_sources) # pyright: ignore[reportCallIssue]
# --- anonymous -------------------------------------------------------------
def test_anonymous_gets_403_on_all_routes(client: TestClient, db: Session) -> None:
r = client.get("/api/git-sources")
assert r.status_code == 403
assert r.json() == {"detail": "admin only"}
r = client.post("/api/git-sources", json={"url": "https://anon.example.com/x.git"})
assert r.status_code == 403
assert r.json() == {"detail": "admin only"}
r = client.delete(f"/api/git-sources/{uuid.uuid4()}")
assert r.status_code == 403
assert r.json() == {"detail": "admin only"}
# Nothing landed in the table.
assert db.execute(text("SELECT count(*) FROM git_sources")).scalar_one() == 0
# --- GET: env fallback -----------------------------------------------------
def test_get_empty_table_with_env_returns_env_rows(
admin_client: TestClient, monkeypatch: pytest.MonkeyPatch
) -> None:
monkeypatch.setattr(
git_sources_api,
"get_settings",
lambda: _settings("https://a.example.com/one.git, git@b.example.com:two.git ,"),
)
r = admin_client.get("/api/git-sources")
assert r.status_code == 200
body = r.json()
assert body["from_env"] is True
# Whitespace-trimmed, empty entries dropped, order preserved; null ids.
assert body["sources"] == [
{"id": None, "url": "https://a.example.com/one.git", "added_at": None},
{"id": None, "url": "git@b.example.com:two.git", "added_at": None},
]
def test_get_empty_table_with_empty_env_returns_empty_list(
admin_client: TestClient, monkeypatch: pytest.MonkeyPatch
) -> None:
monkeypatch.setattr(git_sources_api, "get_settings", lambda: _settings())
r = admin_client.get("/api/git-sources")
assert r.status_code == 200
assert r.json() == {"sources": [], "from_env": True}
def test_get_orders_db_rows_by_added_at_then_id(admin_client: TestClient, db: Session) -> None:
base = datetime.now(UTC) - timedelta(hours=3)
db.add_all(
[
GitSource(url="https://example.com/oldest.git", added_at=base),
GitSource(url="https://example.com/second.git", added_at=base + timedelta(hours=1)),
GitSource(url="https://example.com/newest.git", added_at=base + timedelta(hours=2)),
# Same transaction → identical server-stamped added_at (≈ now,
# after the explicit rows): the id tie-break decides their order.
GitSource(url="https://example.com/tie-a.git"),
GitSource(url="https://example.com/tie-b.git"),
]
)
db.commit()
tie_rows = db.scalars(
select(GitSource).where(GitSource.url.like("https://example.com/tie-%"))
).all()
tie_ids = sorted(row.id for row in tie_rows)
body = admin_client.get("/api/git-sources").json()
assert body["from_env"] is False
assert [s["url"] for s in body["sources"][:3]] == [
"https://example.com/oldest.git",
"https://example.com/second.git",
"https://example.com/newest.git",
]
assert [s["id"] for s in body["sources"][3:]] == [str(i) for i in tie_ids]
# DB rows carry real ids + timestamps (the env shape has neither).
for s in body["sources"]:
assert s["id"] is not None
assert s["added_at"] is not None
# --- POST: create ----------------------------------------------------------
def test_post_creates_trimmed_and_list_stops_using_env(
admin_client: TestClient, monkeypatch: pytest.MonkeyPatch
) -> None:
monkeypatch.setattr(
git_sources_api,
"get_settings",
lambda: _settings("https://env.example.com/env.git"),
)
r = admin_client.post("/api/git-sources", json={"url": " https://new.example.com/repo.git "})
assert r.status_code == 201
body = r.json()
assert body["url"] == "https://new.example.com/repo.git" # trimmed
uuid.UUID(body["id"])
assert body["added_at"] is not None
assert set(body) == {"id", "url", "added_at"}
# The DB row now wins: from_env False, the env URL is gone from the list.
body = admin_client.get("/api/git-sources").json()
assert body["from_env"] is False
assert [s["url"] for s in body["sources"]] == ["https://new.example.com/repo.git"]
def test_post_accepts_accepted_url_shapes(admin_client: TestClient) -> None:
for url in (
"https://github.com/owner/repo.git",
"http://git.local/repo.git",
"ssh://git@example.com/repo.git",
"git@github.com:owner/repo.git",
):
r = admin_client.post("/api/git-sources", json={"url": url})
assert r.status_code == 201, f"{url} must be accepted: {r.text}"
assert r.json()["url"] == url
def test_post_duplicate_url_returns_409_without_echoing_url(
admin_client: TestClient, monkeypatch: pytest.MonkeyPatch
) -> None:
"""The 409 detail is a fixed generic string — URLs may embed
``user:pass@`` credentials (phase 32's masking discipline)."""
monkeypatch.setattr(git_sources_api, "get_settings", lambda: _settings())
url = "https://user:secret@example.com/creds.git"
assert admin_client.post("/api/git-sources", json={"url": url}).status_code == 201
# Same URL with different surrounding whitespace: the trim makes it a
# duplicate too.
r = admin_client.post("/api/git-sources", json={"url": f" {url}\t"})
assert r.status_code == 409
detail = r.json()["detail"]
assert detail == "a git source with this URL already exists"
assert url not in detail
assert "user:secret" not in detail
# Exactly one row stored.
assert len(admin_client.get("/api/git-sources").json()["sources"]) == 1
def test_post_concurrent_insert_backstop_still_409(
admin_client: TestClient, monkeypatch: pytest.MonkeyPatch
) -> None:
"""If the duplicate pre-check misses (a concurrent insert lands
between the check and the commit), the DB unique index still yields
the generic 409 — never a 500."""
monkeypatch.setattr(git_sources_api, "get_settings", lambda: _settings())
url = "https://example.com/backstop.git"
assert admin_client.post("/api/git-sources", json={"url": url}).status_code == 201
real_select = git_sources_api.select
def blind_select(*args: Any, **kwargs: Any) -> Any:
if args and args[0] is GitSource: # the duplicate pre-check
# …now never matches — only the unique index can catch it.
return real_select(GitSource).where(GitSource.url == "zz-never-matches")
return real_select(*args, **kwargs)
monkeypatch.setattr(git_sources_api, "select", blind_select)
r = admin_client.post("/api/git-sources", json={"url": url})
assert r.status_code == 409
detail = r.json()["detail"]
assert detail == "a git source with this URL already exists"
assert url not in detail
def test_post_rejects_invalid_shapes_without_echoing_input(
admin_client: TestClient, db: Session
) -> None:
"""Bad shapes are 422 with a detail that never repeats the submitted
value."""
for bad in ("not a url", "host:repo", "ftp://example.com/x.git"):
r = admin_client.post("/api/git-sources", json={"url": bad})
assert r.status_code == 422, f"{bad!r} must be rejected"
assert bad not in r.text, f"422 detail must not echo the input ({bad!r})"
# Nothing stored.
assert db.execute(text("SELECT count(*) FROM git_sources")).scalar_one() == 0
def test_post_rejects_blank_and_oversized_urls(admin_client: TestClient, db: Session) -> None:
"""Whitespace-only (trim → empty) and >500-char URLs are 422; the
500-char boundary passes."""
assert admin_client.post("/api/git-sources", json={"url": " \t\n "}).status_code == 422
assert admin_client.post("/api/git-sources", json={"url": "x" * 501}).status_code == 422
boundary = "https://" + "x" * 492 # exactly 500 chars
assert len(boundary) == 500
assert admin_client.post("/api/git-sources", json={"url": boundary}).status_code == 201
assert db.execute(text("SELECT count(*) FROM git_sources")).scalar_one() == 1
# --- DB rows win over env ---------------------------------------------------
def test_db_rows_win_over_env(admin_client: TestClient, db: Session, monkeypatch) -> None:
"""Seed a row AND set the env: the GET returns only the DB rows and
``from_env: false`` — the env var is ignored once the table has rows."""
monkeypatch.setattr(
git_sources_api, "get_settings", lambda: _settings("https://env.example.com/env.git")
)
db.add(GitSource(url="https://db.example.com/db.git"))
db.commit()
body = admin_client.get("/api/git-sources").json()
assert body["from_env"] is False
assert [s["url"] for s in body["sources"]] == ["https://db.example.com/db.git"]
for s in body["sources"]:
assert s["id"] is not None
assert s["added_at"] is not None
# --- DELETE -----------------------------------------------------------------
def test_delete_removes_row_and_falls_back_to_env(
admin_client: TestClient, monkeypatch: pytest.MonkeyPatch
) -> None:
monkeypatch.setattr(
git_sources_api,
"get_settings",
lambda: _settings("https://env.example.com/env.git"),
)
created = admin_client.post("/api/git-sources", json={"url": "https://new.example.com/x.git"})
assert created.status_code == 201
assert admin_client.delete(f"/api/git-sources/{created.json()['id']}").status_code == 204
# The table is empty again → the env fallback is live once more.
body = admin_client.get("/api/git-sources").json()
assert body["from_env"] is True
assert body["sources"] == [
{"id": None, "url": "https://env.example.com/env.git", "added_at": None}
]
def test_delete_unknown_id_returns_404(admin_client: TestClient) -> None:
r = admin_client.delete(f"/api/git-sources/{uuid.uuid4()}")
assert r.status_code == 404
assert r.json() == {"detail": "git source not found"}
def test_delete_invalid_id_returns_422(admin_client: TestClient) -> None:
assert admin_client.delete("/api/git-sources/not-a-uuid").status_code == 422
tests/integration/test_import_docs_git.py
+51 -4
View File
@@ -4,9 +4,14 @@ Drives ``scripts.import_docs`` end to end with a fake ``clone_or_pull`` (no
real git, no network) and a recording fake ``import_sources`` (no real
DB), covering:
- ``BOR_GIT_SOURCES`` set → each URL is cloned/pulled into
``BOR_SOURCES_DIR/<repo-name>/`` and exactly those dirs are imported.
- ``--source`` still wins over ``BOR_GIT_SOURCES`` (no git at all).
- Effective git sources set (phase 35: the shared resolver — stubbed
here, keeping this file's no-real-DB style) → each URL is cloned/pulled
into ``BOR_SOURCES_DIR/<repo-name>/`` and exactly those dirs are
imported.
- DB rows win over ``BOR_GIT_SOURCES`` (the resolver's ``db`` origin —
the env list is ignored).
- ``--source`` still wins over git sources (no git at all, no resolver
call).
- No git sources + no ``--source`` → the legacy ``DEFAULT_SOURCES``.
- A failing git sync → exit code 1, an error naming the failing repo on
stderr, and **zero** import attempts.
@@ -91,6 +96,13 @@ def test_resolve_sources_git_urls_cloned_into_sources_dir(
) -> None:
calls, fake = _fake_clone_factory()
monkeypatch.setattr(import_docs, "clone_or_pull", fake)
# Phase 35: resolution goes through the shared resolver (stubbed —
# this file keeps its no-real-DB style); the URLs are the env list.
monkeypatch.setattr(
import_docs,
"effective_git_sources",
lambda db: (["https://host/a/homelab.git", "git@host:user/deploy.git"], "env"),
)
settings = _settings(
git_sources="https://host/a/homelab.git, git@host:user/deploy.git ,",
sources_dir=str(tmp_path / "bor"),
@@ -119,7 +131,34 @@ def test_resolve_sources_cli_source_wins(
assert calls == [] # git is never touched when --source is given
def test_resolve_sources_defaults_when_nothing_configured() -> None:
def test_resolve_sources_db_rows_win_over_env(
monkeypatch: pytest.MonkeyPatch, tmp_path: Path
) -> None:
"""Phase 35: the resolver's ``db`` origin (table has rows) — the
``BOR_GIT_SOURCES`` list must be ignored; only the DB repo is cloned."""
calls, fake = _fake_clone_factory()
monkeypatch.setattr(import_docs, "clone_or_pull", fake)
monkeypatch.setattr(
import_docs,
"effective_git_sources",
lambda db: (["https://db.example/only.git"], "db"),
)
settings = _settings(
git_sources="https://env.example/ignored.git",
sources_dir=str(tmp_path / "bor"),
)
sources = import_docs._resolve_sources(None, settings)
assert sources == [tmp_path / "bor" / "only"]
assert calls == [("https://db.example/only.git", tmp_path / "bor" / "only")]
def test_resolve_sources_defaults_when_nothing_configured(
monkeypatch: pytest.MonkeyPatch,
) -> None:
# Both origins empty (the resolver's ``([], "env")``) → legacy dirs.
monkeypatch.setattr(import_docs, "effective_git_sources", lambda db: ([], "env"))
sources = import_docs._resolve_sources(None, _settings())
assert sources == [p.expanduser() for p in import_docs.DEFAULT_SOURCES]
@@ -137,6 +176,11 @@ def test_main_git_sources_clone_then_import(
sources_dir=str(tmp_path / "bor"),
)
monkeypatch.setattr(import_docs, "get_settings", lambda: settings)
monkeypatch.setattr(
import_docs,
"effective_git_sources",
lambda db: (["https://host/a/homelab.git", "https://host/a/deploy.git"], "env"),
)
calls, fake = _fake_clone_factory()
monkeypatch.setattr(import_docs, "clone_or_pull", fake)
fake_import = FakeImportSources()
@@ -191,6 +235,9 @@ def test_main_git_failure_aborts_before_import(
sources_dir=str(tmp_path / "bor"),
)
monkeypatch.setattr(import_docs, "get_settings", lambda: settings)
monkeypatch.setattr(
import_docs, "effective_git_sources", lambda db: (["https://host/a/bad.git"], "env")
)
def failing_clone(url: str, dest: Path | str) -> Path:
raise GitSyncError(
tests/integration/test_migration_0005.py
+14 -9
View File
@@ -5,12 +5,15 @@ Drives the **real Alembic engine** against the live dev database
``test_migration_0004.py`` (information_schema assertions on the state the
migration must leave):
* upgrade to head → ``kb_overview`` exists with exactly the three columns
The tests target revision ``0005`` explicitly so later migrations
(0006, …) cannot break them.
* upgrade to 0005 → ``kb_overview`` exists with exactly the three columns
the phase locks in (``id INTEGER PK`` default 1, ``content TEXT NOT NULL``
default ``''``, ``updated_at TIMESTAMPTZ NOT NULL`` default ``now()``),
and a bare insert lands the single-row defaults (id=1, content='');
* downgrade to 0004 → the table is gone;
* upgrade to head again → it is back (round-trip).
* upgrade to 0005 again → it is back (round-trip).
The ``alembic`` fixture guarantees the DB ends at head even if a test
fails or the process is interrupted.
@@ -65,14 +68,14 @@ def _version(db: Session) -> str | None:
return db.execute(text("SELECT version_num FROM alembic_version")).scalar()
def test_upgrade_to_head_creates_kb_overview(db: Session, alembic: Config) -> None:
"""Upgrade to head: the single-row table exists with the locked
def test_upgrade_to_0005_creates_kb_overview(db: Session, alembic: Config) -> None:
"""Upgrade 0004 → 0005: the single-row table exists with the locked
column types, nullability, and server defaults."""
command.downgrade(alembic, "0004") # start from the pre-0005 state
assert _version(db) == "0004"
command.upgrade(alembic, "head")
assert _version(db) == "0005", "alembic_version must be at 0005 (head)"
command.upgrade(alembic, "0005")
assert _version(db) == "0005", "alembic_version must be at 0005"
pk = db.execute(
text(
@@ -139,9 +142,11 @@ def test_downgrade_to_0004_drops_table(db: Session, alembic: Config) -> None:
def test_upgrade_round_trip_restores_table(db: Session, alembic: Config) -> None:
"""Upgrade back to head after the downgrade: table + defaults are back."""
command.upgrade(alembic, "head")
assert _version(db) == "0005", "round-trip upgrade must land at 0005 (head)"
"""Downgrade to 0004, then upgrade back to 0005: table + defaults are
back (self-contained — does not rely on a prior test's downgrade)."""
command.downgrade(alembic, "0004")
command.upgrade(alembic, "0005")
assert _version(db) == "0005", "round-trip upgrade must land at 0005"
id_col = _column(db, "id")
assert id_col is not None and id_col[2] == "1", "kb_overview.id must be back with default 1"
tests/integration/test_migration_0006.py
+194
View File
@@ -0,0 +1,194 @@
"""Integration: migration 0006 (git_sources) schema contract.
Drives the **real Alembic engine** against the live dev database
(``podman compose up -d db``), mirroring the style of
``test_migration_0004.py`` / ``test_migration_0005.py`` (information_schema
assertions on the state the migration must leave). The tests target
revision ``0006`` explicitly so later migrations (0007, …) cannot break
them:
* upgrade 0005 → 0006 → ``git_sources`` exists with exactly the three
columns the phase locks in (``id UUID PK``, ``url TEXT NOT NULL``,
``added_at TIMESTAMPTZ NOT NULL`` default ``now()``) plus the unique
index ``uq_git_sources_url`` (duplicate URLs rejected with an
IntegrityError);
* an insert without ``added_at`` gets the server-stamped default;
* downgrade to 0005 → the table (and its index) is gone;
* upgrade back to 0006 → it is back (round-trip).
The ``alembic`` fixture guarantees the DB ends at head even if a test
fails or the process is interrupted.
"""
from __future__ import annotations
from collections.abc import Iterator
from typing import Any
import pytest
from alembic.config import Config
from sqlalchemy import text
from sqlalchemy.exc import IntegrityError
from sqlalchemy.orm import Session
from alembic import command
from app.db import db_available
@pytest.fixture()
def alembic(db: Session) -> Iterator[Config]:
"""Real Alembic config bound to the dev DB (URL from app settings).
Starts at head (repairs an interrupted earlier run); teardown upgrades
to head no matter what happened, so the dev DB is never left below
head.
"""
if not db_available():
pytest.skip("Postgres not reachable — run `podman compose up -d db` first")
cfg = Config() # no alembic.ini file — env.py gets the URL from app config
cfg.set_main_option("script_location", "alembic")
command.upgrade(cfg, "head")
try:
yield cfg
finally:
command.upgrade(cfg, "head")
def _column(db: Session, column: str) -> tuple[Any, ...] | None:
"""(data_type, is_nullable, column_default) for one git_sources column."""
row = db.execute(
text(
"SELECT data_type, is_nullable, column_default"
" FROM information_schema.columns"
" WHERE table_name = 'git_sources' AND column_name = :c"
),
{"c": column},
).fetchone()
return tuple(row) if row is not None else None
def _version(db: Session) -> str | None:
return db.execute(text("SELECT version_num FROM alembic_version")).scalar()
def _unique_url_index(db: Session) -> int:
"""1 iff ``uq_git_sources_url`` exists as a UNIQUE index."""
count: Any = db.execute(
text(
"SELECT count(*) FROM pg_indexes"
" WHERE tablename = 'git_sources' AND indexname = 'uq_git_sources_url'"
" AND indexdef ILIKE 'CREATE UNIQUE%'"
)
).scalar()
assert count is not None, "pg_indexes count must be an int"
return int(count)
def _insert_url(db: Session, url: str) -> None:
db.execute(
text("INSERT INTO git_sources (id, url) VALUES (gen_random_uuid(), :u)"),
{"u": url},
)
db.commit()
def test_upgrade_to_0006_creates_git_sources(db: Session, alembic: Config) -> None:
"""Upgrade 0005 → 0006: the table exists with the locked column types,
nullability, server default, primary key, and unique URL index."""
command.downgrade(alembic, "0005") # start from the pre-0006 state
assert _version(db) == "0005"
command.upgrade(alembic, "0006")
assert _version(db) == "0006", "alembic_version must be at 0006"
pk = db.execute(
text(
"SELECT column_name FROM information_schema.table_constraints tc"
" JOIN information_schema.key_column_usage kcu"
" ON tc.constraint_name = kcu.constraint_name"
" WHERE tc.table_name = 'git_sources' AND tc.constraint_type = 'PRIMARY KEY'"
)
).scalar()
assert pk == "id", "git_sources primary key must be id"
id_col = _column(db, "id")
assert id_col is not None, "git_sources.id is missing"
assert id_col[0] == "uuid", "git_sources.id must be UUID"
assert id_col[1] == "NO", "git_sources.id must be NOT NULL"
url = _column(db, "url")
assert url is not None, "git_sources.url is missing"
assert url[0] == "text", "git_sources.url must be TEXT"
assert url[1] == "NO", "git_sources.url must be NOT NULL"
added = _column(db, "added_at")
assert added is not None, "git_sources.added_at is missing"
assert added[0] == "timestamp with time zone", "git_sources.added_at must be TIMESTAMPTZ"
assert added[1] == "NO", "git_sources.added_at must be NOT NULL"
assert added[2] is not None and "now()" in added[2], (
"git_sources.added_at must have server default now()"
)
assert _unique_url_index(db) == 1, "uq_git_sources_url unique index is missing"
def test_added_at_defaults_to_now(db: Session, alembic: Config) -> None:
"""An insert without ``added_at`` gets the server-stamped default —
the API layer never sets it itself (phase 35 task 02)."""
command.upgrade(alembic, "head")
try:
_insert_url(db, "https://git.example.com/mig-test/added-at.git")
stamped = db.execute(
text(
"SELECT added_at IS NOT NULL FROM git_sources"
" WHERE url = 'https://git.example.com/mig-test/added-at.git'"
)
).scalar()
assert stamped is True, "git_sources.added_at must be stamped by the server"
finally:
db.execute(
text(
"DELETE FROM git_sources"
" WHERE url = 'https://git.example.com/mig-test/added-at.git'"
)
)
db.commit()
def test_duplicate_url_rejected(db: Session, alembic: Config) -> None:
"""The unique index is what the API's 409 relies on: a duplicate URL
raises IntegrityError."""
command.upgrade(alembic, "head")
try:
_insert_url(db, "https://git.example.com/mig-test/dup.git")
with pytest.raises(IntegrityError):
_insert_url(db, "https://git.example.com/mig-test/dup.git")
finally:
db.rollback() # the IntegrityError aborts the open transaction
db.execute(
text("DELETE FROM git_sources WHERE url = 'https://git.example.com/mig-test/dup.git'")
)
db.commit()
def test_downgrade_to_0005_drops_table(db: Session, alembic: Config) -> None:
"""Downgrade to 0005: the table (and its index) is dropped (A13 —
reversible)."""
command.downgrade(alembic, "0005")
assert _version(db) == "0005"
exists = db.execute(text("SELECT to_regclass('public.git_sources') IS NOT NULL")).scalar()
assert exists is False, "git_sources must be dropped by the downgrade"
def test_upgrade_round_trip_restores_table(db: Session, alembic: Config) -> None:
"""Downgrade to 0005, then upgrade back to 0006: table, defaults, and
the unique index are back."""
command.downgrade(alembic, "0005")
command.upgrade(alembic, "0006")
assert _version(db) == "0006", "round-trip upgrade must land at 0006"
added = _column(db, "added_at")
assert added is not None and added[2] is not None and "now()" in added[2], (
"git_sources.added_at must keep its now() default after the round-trip"
)
assert _unique_url_index(db) == 1, "uq_git_sources_url must be back after the round-trip"
tests/integration/test_sync_api.py
+149 -25
View File
@@ -1,17 +1,26 @@
"""Integration: the admin sources-sync API (phase 32, task 01).
"""Integration: the admin sources-sync API (phase 32, task 01; phase 35,
task 03 re-points the URL resolution at the shared resolver).
Covers the in-process sync runner end to end over HTTP: anonymous 403s
on both endpoints; admin idle → 202 → ``success`` with the full
ImportSummary detail; 409 on a double trigger while a run is in flight;
``GitSyncError`` → ``failed`` with the failing repo named and **zero**
import attempts; empty ``BOR_GIT_SOURCES`` → ``failed`` loudly; an
embedding failure → ``failed`` with any credentials masked; the import
always runs with ``prune=True``; and the phase-31 overview trigger is
change-gated (no ``lite`` call on an unchanged KB).
import attempts; empty on *both* origins (``git_sources`` table +
``BOR_GIT_SOURCES``) → ``failed`` loudly; an embedding failure →
``failed`` with any credentials masked; the import always runs with
``prune=True``; and the phase-31 overview trigger is change-gated (no
``lite`` call on an unchanged KB).
Phase 35: the runner resolves the repos through
:func:`app.rag.git_sources.effective_git_sources` — the **real**
resolver against the **real** ``git_sources`` table (truncated around
every test), so DB-over-env and the env fallback go through the actual
indirection; the env list is driven by a fresh ``Settings`` on the
resolver's module (the dev ``.env`` never leaks in).
The git / import / overview layers are monkeypatched in ``app.api.sync``
(same fake style as ``test_import_docs_git.py``) — no real git, no real
DB, no LLM: the runner's state machine and HTTP surface are under test.
(same fake style as ``test_import_docs_git.py``) — no real git, no LLM:
the runner's state machine and HTTP surface are under test.
The admin client is used **as a context manager** on purpose: the
background sync task lives on the app's event loop, so the loop must
@@ -22,6 +31,7 @@ request and would cancel the task on request exit.)
from __future__ import annotations
import asyncio
import logging
import time
from collections.abc import Iterator
from datetime import datetime
@@ -29,11 +39,14 @@ from pathlib import Path
import pytest
from fastapi.testclient import TestClient
from sqlalchemy import text
from sqlalchemy.orm import Session
from app.api import sync as sync_api
from app.config import Settings
from app.main import app as fastapi_app
from app.models import GitSource
from app.rag import git_sources as git_sources_resolver
from app.rag.importer import ImportSummary
from app.rag.llm import EmbeddingError, LLMClient
from scripts.git_sync import GitSyncError
@@ -52,6 +65,18 @@ def _fresh_sync_state() -> Iterator[None]:
sync_api._task = None
@pytest.fixture(autouse=True)
def clean_git_sources(db: Session) -> Iterator[None]:
"""Phase 35: the runner resolves through the real ``git_sources``
table — global state, truncated around every test (the ``db``
fixture skips the file when Postgres is down)."""
db.execute(text("TRUNCATE git_sources"))
db.commit()
yield
db.execute(text("TRUNCATE git_sources"))
db.commit()
@pytest.fixture()
def sync_client() -> Iterator[TestClient]:
"""Context-managed TestClient — one app event loop across requests
@@ -60,9 +85,29 @@ def sync_client() -> Iterator[TestClient]:
yield client
def _settings(git_sources: str = "", sources_dir: str = "~/bor-sources") -> Settings:
"""Fresh settings (no .env file); explicit kwargs beat any env leaks."""
return Settings(_env_file=None, git_sources=git_sources, sources_dir=sources_dir) # pyright: ignore[reportCallIssue]
def _settings(sources_dir: str = "~/bor-sources") -> Settings:
"""Fresh settings (no .env file); explicit kwargs beat any env leaks.
(The ``git_sources`` kwarg is gone with phase 35 — the runner reads
the URLs from the resolver, not from its own settings; the env list
is stubbed on the resolver's module via :func:`_stub_env`.)
"""
return Settings(_env_file=None, sources_dir=sources_dir) # pyright: ignore[reportCallIssue]
def _stub_env(monkeypatch: pytest.MonkeyPatch, git_sources: str = "") -> None:
"""The resolver's env fallback, driven by a fresh ``Settings`` (the
dev ``.env`` never leaks in — the task-02 pattern)."""
monkeypatch.setattr(
git_sources_resolver,
"get_settings",
lambda: Settings(_env_file=None, git_sources=git_sources), # pyright: ignore[reportCallIssue]
)
def _seed(db: Session, url: str) -> None:
db.add(GitSource(url=url))
db.commit()
def _login(client: TestClient) -> None:
@@ -157,13 +202,15 @@ def test_anonymous_gets_403_on_both_endpoints(client: TestClient) -> None:
def test_admin_sync_success_reports_full_detail(
sync_client: TestClient, monkeypatch: pytest.MonkeyPatch, tmp_path: Path
sync_client: TestClient, monkeypatch: pytest.MonkeyPatch, db: Session, tmp_path: Path
) -> None:
repo_url = f"file://{tmp_path / 'repo.git'}"
_seed(db, repo_url) # the phase-35 resolver picks the DB row up
_stub_env(monkeypatch) # env must not matter once the table has a row
monkeypatch.setattr(
sync_api,
"get_settings",
lambda: _settings(git_sources=repo_url, sources_dir=str(tmp_path / "bor")),
lambda: _settings(sources_dir=str(tmp_path / "bor")),
)
clone_calls, fake_clone = _fake_clone()
monkeypatch.setattr(sync_api, "clone_or_pull", fake_clone)
@@ -214,14 +261,16 @@ def test_admin_sync_success_reports_full_detail(
def test_unchanged_kb_skips_overview_refresh(
sync_client: TestClient, monkeypatch: pytest.MonkeyPatch, tmp_path: Path
sync_client: TestClient, monkeypatch: pytest.MonkeyPatch, db: Session, tmp_path: Path
) -> None:
"""Phase-31 trigger is change-gated: added + updated == 0 → no ``lite`` call."""
repo_url = f"file://{tmp_path / 'repo.git'}"
_seed(db, repo_url)
_stub_env(monkeypatch)
monkeypatch.setattr(
sync_api,
"get_settings",
lambda: _settings(git_sources=repo_url, sources_dir=str(tmp_path / "bor")),
lambda: _settings(sources_dir=str(tmp_path / "bor")),
)
_, fake_clone = _fake_clone()
monkeypatch.setattr(sync_api, "clone_or_pull", fake_clone)
@@ -244,13 +293,15 @@ def test_unchanged_kb_skips_overview_refresh(
def test_double_trigger_while_running_returns_409(
sync_client: TestClient, monkeypatch: pytest.MonkeyPatch, tmp_path: Path
sync_client: TestClient, monkeypatch: pytest.MonkeyPatch, db: Session, tmp_path: Path
) -> None:
repo_url = f"file://{tmp_path / 'repo.git'}"
_seed(db, repo_url)
_stub_env(monkeypatch)
monkeypatch.setattr(
sync_api,
"get_settings",
lambda: _settings(git_sources=repo_url, sources_dir=str(tmp_path / "bor")),
lambda: _settings(sources_dir=str(tmp_path / "bor")),
)
_, fake_clone = _fake_clone()
monkeypatch.setattr(sync_api, "clone_or_pull", fake_clone)
@@ -282,13 +333,15 @@ def test_double_trigger_while_running_returns_409(
def test_git_failure_marks_failed_and_skips_import(
sync_client: TestClient, monkeypatch: pytest.MonkeyPatch, tmp_path: Path
sync_client: TestClient, monkeypatch: pytest.MonkeyPatch, db: Session, tmp_path: Path
) -> None:
repo_url = f"file://{tmp_path / 'bad.git'}"
_seed(db, repo_url)
_stub_env(monkeypatch)
monkeypatch.setattr(
sync_api,
"get_settings",
lambda: _settings(git_sources=repo_url, sources_dir=str(tmp_path / "bor")),
lambda: _settings(sources_dir=str(tmp_path / "bor")),
)
def failing_clone(url: str, dest: Path | str) -> Path:
@@ -322,11 +375,14 @@ def test_git_failure_marks_failed_and_skips_import(
def test_no_git_sources_configured_fails_loudly(
sync_client: TestClient, monkeypatch: pytest.MonkeyPatch, tmp_path: Path
) -> None:
"""Both origins empty (the truncate fixture + a blank env) → the
fail-loud error names *both* (phase 35)."""
# Whitespace-only is just as unconfigured as empty.
_stub_env(monkeypatch, " , ")
monkeypatch.setattr(
sync_api,
"get_settings",
# Whitespace-only is just as unconfigured as empty.
lambda: _settings(git_sources=" , ", sources_dir=str(tmp_path / "bor")),
lambda: _settings(sources_dir=str(tmp_path / "bor")),
)
clone_calls, fake_clone = _fake_clone()
monkeypatch.setattr(sync_api, "clone_or_pull", fake_clone)
@@ -337,19 +393,87 @@ def test_no_git_sources_configured_fails_loudly(
assert sync_client.post("/api/sync").status_code == 202
body = _poll(sync_client, "failed")
assert body["error"] == "no git sources configured (BOR_GIT_SOURCES)"
assert body["error"] == (
"no git sources configured (git_sources table empty and BOR_GIT_SOURCES unset)"
)
assert clone_calls == [] # git is never touched
assert fake_import.sources == []
def test_import_error_is_reported_with_credentials_masked(
sync_client: TestClient, monkeypatch: pytest.MonkeyPatch, tmp_path: Path
def test_db_rows_win_over_env(
sync_client: TestClient,
monkeypatch: pytest.MonkeyPatch,
db: Session,
tmp_path: Path,
caplog: pytest.LogCaptureFixture,
) -> None:
repo_url = f"file://{tmp_path / 'repo.git'}"
"""Phase 35: a stored row beats ``BOR_GIT_SOURCES`` — only the DB URL
is cloned, and the started log names the origin."""
db_url = "https://db.example.com/managed.git"
_seed(db, db_url)
_stub_env(monkeypatch, "https://env.example.com/ignored.git")
monkeypatch.setattr(
sync_api,
"get_settings",
lambda: _settings(git_sources=repo_url, sources_dir=str(tmp_path / "bor")),
lambda: _settings(sources_dir=str(tmp_path / "bor")),
)
clone_calls, fake_clone = _fake_clone()
monkeypatch.setattr(sync_api, "clone_or_pull", fake_clone)
fake_import = FakeImportSources(ImportSummary(files=1, added=1))
monkeypatch.setattr(sync_api, "import_sources", fake_import)
monkeypatch.setattr(sync_api, "regenerate_overview", FakeOverview(ok=True))
_login(sync_client)
with caplog.at_level(logging.INFO, logger="app.api.sync"):
assert sync_client.post("/api/sync").status_code == 202
body = _poll(sync_client, "success")
assert clone_calls == [(db_url, tmp_path / "bor" / "managed")]
assert fake_import.sources == [[tmp_path / "bor" / "managed"]]
assert "env.example.com" not in str(body) # the env URL never reaches the UI
assert any("sync: started repos=1 origin=db" in r.getMessage() for r in caplog.records)
def test_env_fallback_when_table_empty(
sync_client: TestClient,
monkeypatch: pytest.MonkeyPatch,
tmp_path: Path,
caplog: pytest.LogCaptureFixture,
) -> None:
"""Phase 35: with the table empty (the truncate fixture), the
``BOR_GIT_SOURCES`` list is what gets cloned — origin ``env``."""
env_url = "https://env.example.com/fallback.git"
_stub_env(monkeypatch, env_url)
monkeypatch.setattr(
sync_api,
"get_settings",
lambda: _settings(sources_dir=str(tmp_path / "bor")),
)
clone_calls, fake_clone = _fake_clone()
monkeypatch.setattr(sync_api, "clone_or_pull", fake_clone)
fake_import = FakeImportSources(ImportSummary(files=1, added=1))
monkeypatch.setattr(sync_api, "import_sources", fake_import)
monkeypatch.setattr(sync_api, "regenerate_overview", FakeOverview(ok=True))
_login(sync_client)
with caplog.at_level(logging.INFO, logger="app.api.sync"):
assert sync_client.post("/api/sync").status_code == 202
_poll(sync_client, "success")
assert clone_calls == [(env_url, tmp_path / "bor" / "fallback")]
assert any("sync: started repos=1 origin=env" in r.getMessage() for r in caplog.records)
def test_import_error_is_reported_with_credentials_masked(
sync_client: TestClient, monkeypatch: pytest.MonkeyPatch, db: Session, tmp_path: Path
) -> None:
repo_url = f"file://{tmp_path / 'repo.git'}"
_seed(db, repo_url)
_stub_env(monkeypatch)
monkeypatch.setattr(
sync_api,
"get_settings",
lambda: _settings(sources_dir=str(tmp_path / "bor")),
)
_, fake_clone = _fake_clone()
monkeypatch.setattr(sync_api, "clone_or_pull", fake_clone)
tests/unit/test_git_sources.py
+110
View File
@@ -0,0 +1,110 @@
"""Unit: the shared git-source resolver (phase 35, task 03).
``effective_git_sources`` is driven with a stubbed session (no Postgres)
and a fresh ``Settings(_env_file=None)`` env (monkeypatched into the
resolver's module — the dev ``.env`` never leaks in, same pattern as the
integration suites): DB rows win in ``(added_at, id)`` order (env
ignored), the ``BOR_GIT_SOURCES`` list is a fallback only while the
table is empty (phase-28 CSV parse reused), and both-empty yields
``([], "env")`` so the callers keep their fail-loud behavior.
"""
from __future__ import annotations
import re
from typing import Any
import pytest
from app.config import Settings
from app.models import GitSource
from app.rag import git_sources as resolver
class _FakeScalars:
"""The ``.scalars(stmt).all()`` tail of the resolver's query."""
def __init__(self, rows: list[GitSource]) -> None:
self._rows = rows
def all(self) -> list[GitSource]:
return self._rows
class _FakeSession:
"""Just enough of a SQLAlchemy session for the resolver (test_steering
pattern). Records the statement so the ordering can be asserted."""
def __init__(self, rows: list[GitSource]) -> None:
self._rows = rows
self.statements: list[Any] = []
def scalars(self, stmt: Any) -> _FakeScalars:
self.statements.append(stmt)
return _FakeScalars(self._rows)
def _stub_env(monkeypatch: pytest.MonkeyPatch, git_sources: str) -> None:
"""Point the resolver's env fallback at a fresh Settings (no .env)."""
monkeypatch.setattr(
resolver,
"get_settings",
lambda: Settings(_env_file=None, git_sources=git_sources), # pyright: ignore[reportCallIssue]
)
# --- the three branches ----------------------------------------------------
def test_db_rows_win_over_env(monkeypatch: pytest.MonkeyPatch) -> None:
"""Seeded table + env set → the DB list, origin ``db`` (env ignored)."""
_stub_env(monkeypatch, "https://env.example/ignored.git")
session = _FakeSession([GitSource(url="https://db.example/a.git"), GitSource(url="https://db.example/b.git")])
urls, origin = resolver.effective_git_sources(session) # pyright: ignore[reportArgumentType]
assert urls == ["https://db.example/a.git", "https://db.example/b.git"]
assert origin == "db"
def test_env_fallback_while_table_empty(monkeypatch: pytest.MonkeyPatch) -> None:
"""Empty table + env set → the env list, origin ``env``.
The CSV parse is ``Settings.git_source_list`` itself (phase 28):
whitespace-trimmed, empty entries dropped, order preserved.
"""
_stub_env(monkeypatch, " https://env.example/one.git , ,git@env.example:two.git ")
session = _FakeSession([])
urls, origin = resolver.effective_git_sources(session) # pyright: ignore[reportArgumentType]
assert urls == ["https://env.example/one.git", "git@env.example:two.git"]
assert origin == "env"
def test_both_empty_returns_empty_env(monkeypatch: pytest.MonkeyPatch) -> None:
"""Empty table + unset env → ``([], "env")`` — the callers fail loudly."""
_stub_env(monkeypatch, " , ") # whitespace-only is just as unconfigured as empty
session = _FakeSession([])
urls, origin = resolver.effective_git_sources(session) # pyright: ignore[reportArgumentType]
assert urls == []
assert origin == "env"
# --- the ordering ----------------------------------------------------------
def test_db_rows_ordered_by_added_at_then_id(monkeypatch: pytest.MonkeyPatch) -> None:
"""The statement orders by ``(added_at, id)`` — oldest first, the id
tie-break deciding same-timestamp inserts (matches the API's GET)."""
_stub_env(monkeypatch, "")
session = _FakeSession([GitSource(url="https://db.example/a.git")])
resolver.effective_git_sources(session) # pyright: ignore[reportArgumentType]
assert len(session.statements) == 1
compiled = str(session.statements[0].compile(compile_kwargs={"literal_binds": True}))
assert re.search(
r"ORDER BY\s+git_sources\.added_at ASC,\s+git_sources\.id ASC", compiled
), compiled