phase: 121_git_source_tokens

**Phase 121 final verification pass — all green** (all 4 tasks already in `complete/`; verified, no defects found, no changes needed)

- Verified implementation vs phase design: migration `0021` (reversible, round-tripped via `alembic downgrade base` + `upgrade head` → head `0021`), `GitSource.token` column, `normalize_credential`/`clone_url_for`/`sanitize_url`, clone callers switched (`sync.py`, `import_docs.py`), masked token fields in add form + editor, `extra="forbid"` output shapes
- Tests: `uv run pytest` → 2662 passed, 0 failed (exit 0); `uv run pytest --cov=app --cov-report=term-missing` → TOTAL **99%** (≥90% gate)
- Lint/types: `uv run ruff check .` → All checks passed; `uv run pyright` → 0 errors, 0 warnings
- E2E in isolation: `uv run pytest tests/e2e/test_git_source_tokens.py -v --no-cov` → **4 passed**

Completion criteria:
1. Private repo (UI add or pasted embedded-token URL) clones with injected token; token absent from every API response, page text, title attr, and full HTML — **PASS** (integration raw-JSON assertions + E2E `_assert_token_nowhere`)
2. Legacy embedded-token rows still clone from stored URL; output sanitized — **PASS** (`test_sync_legacy_row_clones_with_original_stored_url`, `test_get_masks_legacy_embedded_token_row`, env-fallback masking)
3. Public/local sources byte-identical — **PASS** (verbatim-URL + no-userinfo-unchanged tests)
4. pytest / coverage / ruff / pyright — **PASS** (see above)
5. Commit + phase move — harness responsibility; task files already in `complete/`, changes left in working tree (no commit made, per protocol)

Notable: no deviations; DB left at head, functional. Next pending phase: **122_image_documents** (then 123_chat_image_questions).

.agents/phases/complete/121_git_source_tokens/00_phase.md

@@ -0,0 +1,48 @@
+# Phase 121 — Private git sources: a token that never reaches the UI or the API
+
+**Source:** `TODO.md` L5 — "Need a way to add private repos without exposing the token in the UI (like when adding an https repo `https://myuser:ghp_xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx@github.com/myuser/my-private-repo.git`)"
+**Story:** n/a (feature request; extends the phase-28/35/38 git/local sources and phase-89 per-source settings assets).
+**Context:** `app/models.py:245` — `GitSource`: `url: Text UNIQUE NOT NULL`, `kind` ("git"/"local"), `path`, `ignore_paths` (JSONB), `include_hidden`, `added_at` — no token column. `app/schemas.py` — `GitSourceIn` (L561: `kind`, `url` min 1/max 500, `path`, `ignore_paths`, `include_hidden`; `_trim_url` before-validator L594), `GitSourceOut` (L605: `url: str`), `GitSourceRow` (L626: `url`), `GitSourcePatchIn` (L651). `app/api/git_sources.py` — `URL_RE = ^(?:https?://|ssh://|git@)` (L195; a prefix match, so `user:token@` URLs pass); POST validates the prefix (L344–345) and duplicates via `GitSource.url == url` (L349); GET list / GET single return `row.url` RAW (L235, L250, L297, L423) — an embedded token is echoed to any browser (the leak); the local-source upload endpoint (L432). `app/api/sync.py:296` — `clone_or_pull(row.url, sources_root / repo_name(row.url))`; `scripts/git_sync.py` — `clone_or_pull`, `repo_name`; `scripts/import_docs.py::_resolve_sources` — the CLI's second clone caller (both consume `app.rag.git_sources.effective_sources`, L27). `frontend/assets/git-sources.js` — add form `#git-source-url` (L241; submit `body: (url) => ({ url })` L945); every display site renders `s.url` raw (list cell L395–406 incl. the `title` attr, delete row L533, edit modal L697, ignore-list context L763/L830). `alembic/` — migrations.
+
+## Objective
+Private repos are added with a bare URL plus an optional MASKED token field. The token lives in a dedicated DB column, is injected only into the clone URL at sync/clone time, and is absent from every API response and UI surface — including legacy rows that already embed the token in `url` (those are sanitized on output but keep working).
+
+## Dependencies
+- `120_failed_turn_retry` (todo) — pipeline predecessor (execution order) only; no code dependency.
+- Code dependencies (all complete): phase 35/38 `GitSource` kinds + `effective_sources`, phase 89 per-source settings (the PATCH-field precedent), phase 28 `clone_or_pull`/`repo_name`.
+
+## Design (shared by all tasks — the executor reads this, not the chat)
+- **Storage (task 01, LOCKED A2):** `GitSource.token` — `Text NULL` (NULL = public/no credential; **plaintext by necessity** — the repo must remain cloneable, so the raw credential must be recoverable; the Postgres DB is the trusted store and is never served to the UI; there is deliberately no external secrets backend). `GitSourceIn.token: str | None = None` (max 500, trimmed); `GitSourcePatchIn.token: str | None = None` — PATCH semantics: **absent/None = no change, non-empty = replace, empty string = clear** (the UI offers replace; clear exists for API completeness). `GitSourceOut`/`GitSourceRow` gain NO token field — no response shape ever carries it (LOCKED A2).
+- **Normalization (task 02):** on POST (and PATCH when a new url/token arrives), the server normalizes: if the incoming URL contains userinfo (`user:pass@host`, only for `https?://` URLs — ssh/`git@` carry no userinfo), the **userinfo is stripped** for storage and the embedded credential is moved into `token` — UNLESS the caller also sent an explicit `token` field, which WINS (explicit beats embedded). Pasting the old-style `https://user:ghp_…@github.com/x/y.git` URL still works and ends up token-column-clean. The duplicate check (L349) runs on the NORMALIZED bare URL, so the same repo with a different token is still the same source (409, not a second row).
+- **Clone-time credential (task 02):** `clone_url_for(row) -> str` in `app/rag/git_sources.py` (next to `effective_sources`): `row.url` unchanged when `token` is NULL; otherwise inject `https://x-access-token:<token>@<host>/<path>` (https rows only — a token on a non-https row is a no-op with a warning log). `repo_name` keeps operating on the bare `row.url`. Callers switch from `row.url` to `clone_url_for(row)`: `app/api/sync.py:296` and `scripts/import_docs.py::_resolve_sources` (both already import from `app.rag.git_sources`).
+- **Output sanitization (task 02, LOCKED A2):** every API surface that returns a git URL runs it through `sanitize_url(url)` (new, in `app/rag/git_sources.py`): strips the userinfo component (`https://…@host/…` → `https://host/…`), leaves ssh/`git@`/local paths untouched. Applied to `GitSourceOut.url` / `GitSourceRow.url` construction (GET list L235/L297, GET single, the `BOR_GIT_SOURCES` env fallback rows L250 — env rows can embed tokens too) and to any sync-status field echoing a repo URL (grep for `url=` in the sync responses). Belt-and-braces for legacy embedded-token rows whose credential is NOT in the `token` column: their DB value is untouched (the clone still authenticates from the stored URL) but no API/UI output ever shows the credential.
+- **UI (task 03):** the add form gains a second field — a masked `<input type="password" id="git-source-token">`, optional, labelled "Token (private repos)" with a visible "optional" hint; submit sends `{ url, token }` (token omitted when blank). The edit modal mirrors it with placeholder "leave blank to keep the current token" (blank → omit from PATCH = no change). Every display site keeps rendering `s.url` — now bare by server sanitization, so list cells, `title` attributes, the delete row, and the ignore-list context become token-free with no per-site change. No new CSS beyond reusing the existing form-field styles (the theme's input treatment).
+- **NOT touched:** local-kind sources (no URL credential), the `BOR_GIT_SOURCES` env parsing (its rows are sanitized on OUTPUT only), the upload endpoint, sync scheduling, and the Sources page layout.
+
+## Tasks
+1. `01_token_storage.md` — migration + `GitSource.token` + input schemas (`GitSourceIn`/`GitSourcePatchIn`); no token in any output shape.
+2. `02_clone_url_and_sanitization.md` — URL/token normalization on write, `clone_url_for` at clone time, `sanitize_url` on every output.
+3. `03_ui_token_field.md` — masked token field in the add form + edit modal; display stays `s.url` (now bare).
+4. `04_token_tests.md` — unit + integration + isolated E2E `test_git_source_tokens.py`.
+
+## Testing & Quality
+- Unit: `tests/unit/test_git_source_token.py` (new, task 04) — `sanitize_url` (https userinfo stripped, ssh/git@/local untouched, no-userinfo unchanged), `clone_url_for` (NULL token → bare URL; token → injected; non-https token → bare + no crash), normalization (embedded token moved to the column when no explicit token; explicit token wins; duplicate on bare URL).
+- Integration: `tests/integration/test_git_sources_api.py` (extend) — POST with `token` → GET list/single responses contain the token NOWHERE (assert on the raw JSON text) and show the bare URL; POST with an old-style embedded-token URL → stored bare + token column populated, responses clean; PATCH token replace/clear semantics; the sync flow builds the clone URL with the injected token (mock `clone_or_pull`).
+- E2E: `tests/e2e/test_git_source_tokens.py` (new, task 04) — isolated run per AGENTS.md §4: add a private repo through the Sources UI (bare URL + token) → the list row shows the bare URL, the token is absent from the page text, the `title` attribute, and `GET /api/git-sources` JSON; edit the row (blank token) → no 4xx, token kept.
+- Coverage: **>90%** on `app/` (validate.sh gate).
+
+## Completion Criteria
+- [ ] A private repo added via the UI (or a pasted embedded-token URL) syncs/clones fine, and its token appears in NO API response, NO page text, and NO attribute.
+- [ ] Legacy embedded-token rows (pre-phase) still clone, and their API/UI output is token-free.
+- [ ] Public repos and local sources behave byte-identically to before.
+- [ ] `uv run pytest` green; `uv run pytest --cov=app --cov-report=term-missing` TOTAL >90%; `uv run ruff check . && uv run pyright` clean.
+- [ ] One `--no-gpg-sign` commit; phase dir moved to `.agents/phases/complete/` by the pipeline gate.
+
+## Locked decisions
+- **A2 — the token is stored PLAINTEXT in a dedicated `GitSource.token` column (cloneability requires the raw credential; no external secrets backend), is NEVER returned by any API shape, and legacy embedded-token URLs are sanitized on output while keeping their stored value for clones (owner-confirmed 2026-09-24, roadmap confirmation).**
+- **A6 — pasting an old-style embedded-token URL is accepted and normalized (userinfo → `token` column); an explicit `token` field wins over an embedded one (owner-confirmed: same confirmation — the proposed design).**
+
+## Commit
+```bash
+git add app/ alembic/ frontend/ scripts/ tests/ .agents/phases/ && git commit --no-gpg-sign -m "feat(sources): add private git repos with a masked token that never reaches the UI or API"
+```