brain-of-reese/.agents/phases/todo/121_git_source_tokens/00_phase.md

Phase 121 — Private git sources: a token that never reaches the UI or the API

Source: TODO.md L5 — "Need a way to add private repos without exposing the token in the UI (like when adding an https repo https://myuser:ghp_xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx@github.com/myuser/my-private-repo.git)" Story: n/a (feature request; extends the phase-28/35/38 git/local sources and phase-89 per-source settings assets). Context: app/models.py:245 — GitSource: url: Text UNIQUE NOT NULL, kind ("git"/"local"), path, ignore_paths (JSONB), include_hidden, added_at — no token column. app/schemas.py — GitSourceIn (L561: kind, url min 1/max 500, path, ignore_paths, include_hidden; _trim_url before-validator L594), GitSourceOut (L605: url: str), GitSourceRow (L626: url), GitSourcePatchIn (L651). app/api/git_sources.py — URL_RE = ^(?:https?://|ssh://|git@) (L195; a prefix match, so user:token@ URLs pass); POST validates the prefix (L344–345) and duplicates via GitSource.url == url (L349); GET list / GET single return row.url RAW (L235, L250, L297, L423) — an embedded token is echoed to any browser (the leak); the local-source upload endpoint (L432). app/api/sync.py:296 — clone_or_pull(row.url, sources_root / repo_name(row.url)); scripts/git_sync.py — clone_or_pull, repo_name; scripts/import_docs.py::_resolve_sources — the CLI's second clone caller (both consume app.rag.git_sources.effective_sources, L27). frontend/assets/git-sources.js — add form #git-source-url (L241; submit body: (url) => ({ url }) L945); every display site renders s.url raw (list cell L395–406 incl. the title attr, delete row L533, edit modal L697, ignore-list context L763/L830). alembic/ — migrations.

Objective

Private repos are added with a bare URL plus an optional MASKED token field. The token lives in a dedicated DB column, is injected only into the clone URL at sync/clone time, and is absent from every API response and UI surface — including legacy rows that already embed the token in url (those are sanitized on output but keep working).

Dependencies

• 120_failed_turn_retry (todo) — pipeline predecessor (execution order) only; no code dependency.
• Code dependencies (all complete): phase 35/38 GitSource kinds + effective_sources, phase 89 per-source settings (the PATCH-field precedent), phase 28 clone_or_pull/repo_name.

Design (shared by all tasks — the executor reads this, not the chat)

• Storage (task 01, LOCKED A2): GitSource.token — Text NULL (NULL = public/no credential; plaintext by necessity — the repo must remain cloneable, so the raw credential must be recoverable; the Postgres DB is the trusted store and is never served to the UI; there is deliberately no external secrets backend). GitSourceIn.token: str | None = None (max 500, trimmed); GitSourcePatchIn.token: str | None = None — PATCH semantics: absent/None = no change, non-empty = replace, empty string = clear (the UI offers replace; clear exists for API completeness). GitSourceOut/GitSourceRow gain NO token field — no response shape ever carries it (LOCKED A2).
• Normalization (task 02): on POST (and PATCH when a new url/token arrives), the server normalizes: if the incoming URL contains userinfo (user:pass@host, only for https?:// URLs — ssh/git@ carry no userinfo), the userinfo is stripped for storage and the embedded credential is moved into token — UNLESS the caller also sent an explicit token field, which WINS (explicit beats embedded). Pasting the old-style https://user:ghp_…@github.com/x/y.git URL still works and ends up token-column-clean. The duplicate check (L349) runs on the NORMALIZED bare URL, so the same repo with a different token is still the same source (409, not a second row).
• Clone-time credential (task 02): clone_url_for(row) -> str in app/rag/git_sources.py (next to effective_sources): row.url unchanged when token is NULL; otherwise inject https://x-access-token:<token>@<host>/<path> (https rows only — a token on a non-https row is a no-op with a warning log). repo_name keeps operating on the bare row.url. Callers switch from row.url to clone_url_for(row): app/api/sync.py:296 and scripts/import_docs.py::_resolve_sources (both already import from app.rag.git_sources).
• Output sanitization (task 02, LOCKED A2): every API surface that returns a git URL runs it through sanitize_url(url) (new, in app/rag/git_sources.py): strips the userinfo component (https://…@host/… → https://host/…), leaves ssh/git@/local paths untouched. Applied to GitSourceOut.url / GitSourceRow.url construction (GET list L235/L297, GET single, the BOR_GIT_SOURCES env fallback rows L250 — env rows can embed tokens too) and to any sync-status field echoing a repo URL (grep for url= in the sync responses). Belt-and-braces for legacy embedded-token rows whose credential is NOT in the token column: their DB value is untouched (the clone still authenticates from the stored URL) but no API/UI output ever shows the credential.
• UI (task 03): the add form gains a second field — a masked <input type="password" id="git-source-token">, optional, labelled "Token (private repos)" with a visible "optional" hint; submit sends { url, token } (token omitted when blank). The edit modal mirrors it with placeholder "leave blank to keep the current token" (blank → omit from PATCH = no change). Every display site keeps rendering s.url — now bare by server sanitization, so list cells, title attributes, the delete row, and the ignore-list context become token-free with no per-site change. No new CSS beyond reusing the existing form-field styles (the theme's input treatment).
• NOT touched: local-kind sources (no URL credential), the BOR_GIT_SOURCES env parsing (its rows are sanitized on OUTPUT only), the upload endpoint, sync scheduling, and the Sources page layout.

Tasks

1. 01_token_storage.md — migration + GitSource.token + input schemas (GitSourceIn/GitSourcePatchIn); no token in any output shape.
2. 02_clone_url_and_sanitization.md — URL/token normalization on write, clone_url_for at clone time, sanitize_url on every output.
3. 03_ui_token_field.md — masked token field in the add form + edit modal; display stays s.url (now bare).
4. 04_token_tests.md — unit + integration + isolated E2E test_git_source_tokens.py.

Testing & Quality

• Unit: tests/unit/test_git_source_token.py (new, task 04) — sanitize_url (https userinfo stripped, ssh/git@/local untouched, no-userinfo unchanged), clone_url_for (NULL token → bare URL; token → injected; non-https token → bare + no crash), normalization (embedded token moved to the column when no explicit token; explicit token wins; duplicate on bare URL).
• Integration: tests/integration/test_git_sources_api.py (extend) — POST with token → GET list/single responses contain the token NOWHERE (assert on the raw JSON text) and show the bare URL; POST with an old-style embedded-token URL → stored bare + token column populated, responses clean; PATCH token replace/clear semantics; the sync flow builds the clone URL with the injected token (mock clone_or_pull).
• E2E: tests/e2e/test_git_source_tokens.py (new, task 04) — isolated run per AGENTS.md §4: add a private repo through the Sources UI (bare URL + token) → the list row shows the bare URL, the token is absent from the page text, the title attribute, and GET /api/git-sources JSON; edit the row (blank token) → no 4xx, token kept.
• Coverage: >90% on app/ (validate.sh gate).

Completion Criteria

• A private repo added via the UI (or a pasted embedded-token URL) syncs/clones fine, and its token appears in NO API response, NO page text, and NO attribute.
• Legacy embedded-token rows (pre-phase) still clone, and their API/UI output is token-free.
• Public repos and local sources behave byte-identically to before.
• uv run pytest green; uv run pytest --cov=app --cov-report=term-missing TOTAL >90%; uv run ruff check . && uv run pyright clean.
• One --no-gpg-sign commit; phase dir moved to .agents/phases/complete/ by the pipeline gate.

Locked decisions

• A2 — the token is stored PLAINTEXT in a dedicated GitSource.token column (cloneability requires the raw credential; no external secrets backend), is NEVER returned by any API shape, and legacy embedded-token URLs are sanitized on output while keeping their stored value for clones (owner-confirmed 2026-09-24, roadmap confirmation).
• A6 — pasting an old-style embedded-token URL is accepted and normalized (userinfo → token column); an explicit token field wins over an embedded one (owner-confirmed: same confirmation — the proposed design).

Commit

git add app/ alembic/ frontend/ scripts/ tests/ .agents/phases/ && git commit --no-gpg-sign -m "feat(sources): add private git repos with a masked token that never reaches the UI or API"