phase: 121_git_source_tokens

**Phase 121 final verification pass — all green** (all 4 tasks already in `complete/`; verified, no defects found, no changes needed)

- Verified implementation vs phase design: migration `0021` (reversible, round-tripped via `alembic downgrade base` + `upgrade head` → head `0021`), `GitSource.token` column, `normalize_credential`/`clone_url_for`/`sanitize_url`, clone callers switched (`sync.py`, `import_docs.py`), masked token fields in add form + editor, `extra="forbid"` output shapes
- Tests: `uv run pytest` → 2662 passed, 0 failed (exit 0); `uv run pytest --cov=app --cov-report=term-missing` → TOTAL **99%** (≥90% gate)
- Lint/types: `uv run ruff check .` → All checks passed; `uv run pyright` → 0 errors, 0 warnings
- E2E in isolation: `uv run pytest tests/e2e/test_git_source_tokens.py -v --no-cov` → **4 passed**

Completion criteria:
1. Private repo (UI add or pasted embedded-token URL) clones with injected token; token absent from every API response, page text, title attr, and full HTML — **PASS** (integration raw-JSON assertions + E2E `_assert_token_nowhere`)
2. Legacy embedded-token rows still clone from stored URL; output sanitized — **PASS** (`test_sync_legacy_row_clones_with_original_stored_url`, `test_get_masks_legacy_embedded_token_row`, env-fallback masking)
3. Public/local sources byte-identical — **PASS** (verbatim-URL + no-userinfo-unchanged tests)
4. pytest / coverage / ruff / pyright — **PASS** (see above)
5. Commit + phase move — harness responsibility; task files already in `complete/`, changes left in working tree (no commit made, per protocol)

Notable: no deviations; DB left at head, functional. Next pending phase: **122_image_documents** (then 123_chat_image_questions).

app/api/git_sources.py

@@ -49,11 +49,16 @@ embeddings) committed first, then the app-managed on-disk dir). The
 whole router sits behind :func:`app.core.auth.require_admin` —
 anonymous callers get 403 on every route.
 
-No credential-echo path: git URLs may embed ``user:pass@`` (phase 32's
-masking discipline), so every git 409/422 detail is a fixed generic
-string that never repeats the submitted URL. Local paths are not
-secrets — the local 422/409 details name the (expanded) path so the
-owner sees exactly which directory failed.
+No credential-echo path (phase 32's masking discipline, extended by
+phase 121 — LOCKED A2): git URLs may embed ``user:pass@``, so (a)
+every git 409/422 detail is a fixed generic string that never repeats
+the submitted URL, and (b) every URL that LEAVES the API is masked
+through :func:`app.rag.git_sources.sanitize_url` before it enters a
+response (DB rows, env-fallback rows, POST 201, PATCH 200) — a legacy
+row whose credential is still embedded in the stored ``url`` clones
+fine (the stored value is untouched) but its API/UI output is
+bare. Local paths are not secrets — the local 422/409 details name
+the (expanded) path so the owner sees exactly which directory failed.
 
 Scope boundary (phase locked decisions): the CRUD routes do NOT
 clone or import anything — the existing Sync button performs that, and
@@ -110,6 +115,7 @@ from app.rag.archive_upload import (
     swap_in,
     unpack_archive,
 )
+from app.rag.git_sources import normalize_credential, sanitize_url
 from app.rag.importer import normalize_ignore_path
 from app.rag.llm import LLMClient
 from app.rag.overview import regenerate_overview
@@ -220,6 +226,11 @@ def list_git_sources(
     git-only) with null ``id``/``added_at``, ``ignore_paths: []`` and
     ``include_hidden: False`` (no DB row to store a list or a flag on),
     and ``from_env: true``.
+
+    Every URL is masked on the way out (phase 121, LOCKED A2 —
+    :func:`sanitize_url`): an env value or a legacy stored URL may
+    embed ``user:pass@`` — the env value and the DB value are
+    untouched, only the response is bare.
     """
     rows = db.scalars(
         select(GitSource).order_by(GitSource.added_at.asc(), GitSource.id.asc())
@@ -232,7 +243,7 @@ def list_git_sources(
                 GitSourceRow(
                     id=row.id,
                     kind=cast(Literal["git", "local"], row.kind),
-                    url=row.url,
+                    url=sanitize_url(row.url),  # phase 121: never echo userinfo
                     path=row.path,
                     added_at=row.added_at,
                     ignore_paths=row.ignore_paths or [],
@@ -247,7 +258,7 @@ def list_git_sources(
             GitSourceRow(
                 id=None,
                 kind="git",
-                url=url,
+                url=sanitize_url(url),  # phase 121: an env URL can embed a token
                 path=None,
                 added_at=None,
                 ignore_paths=[],
@@ -290,11 +301,20 @@ def create_git_source(
     ``include_hidden`` (phase 105) — optional, both kinds: absent →
     stored ``False`` (A4), present → stored as sent; the stored flag is
     what is reported.
+
+    ``token`` (phase 121, LOCKED A2) — the masked private-repo
+    credential: write-only, stored in the dedicated column, never
+    echoed (the response has no token field by contract). Git rows
+    are normalized on the way in (``normalize_credential``): an
+    old-style embedded ``user:pass@`` URL is stored bare with the
+    credential in the token column, an explicit ``token`` wins over
+    the embedded one (LOCKED A6), and the duplicate check runs on the
+    bare URL.
     """
     row = _create_git_row(payload, db) if payload.kind == "git" else _create_local_row(payload, db)
     return GitSourceOut(
         id=row.id,
-        url=row.url,
+        url=sanitize_url(row.url),  # phase 121: the output mask, always
         added_at=row.added_at,
         ignore_paths=row.ignore_paths,
         include_hidden=row.include_hidden,
@@ -346,6 +366,13 @@ def _create_git_row(payload: GitSourceIn, db: Session) -> GitSource:
         raise HTTPException(
             status_code=422, detail="not a valid git URL (expected https://, ssh:// or git@…)"
         )
+    # Phase 121 (task 02, LOCKED A6): normalize the credential — an
+    # old-style embedded ``user:pass@`` URL is stored BARE and the
+    # embedded credential moves to the token column; an explicit
+    # ``token`` field wins over the embedded one. The duplicate check
+    # below runs on the BARE URL, so the same repo pasted with a
+    # different credential is the same source (409, not a second row).
+    url, effective_token = normalize_credential(url, payload.token)
     if db.scalar(select(GitSource).where(GitSource.url == url)) is not None:
         raise HTTPException(status_code=409, detail="a git source with this URL already exists")
     return _commit_new(
@@ -354,6 +381,7 @@ def _create_git_row(payload: GitSourceIn, db: Session) -> GitSource:
             kind="git",
             ignore_paths=_validate_ignore_paths(payload.ignore_paths),
             include_hidden=bool(payload.include_hidden),
+            token=effective_token or None,
         ),
         "a git source with this URL already exists",
         db,
@@ -380,7 +408,10 @@ def _create_local_row(payload: GitSourceIn, db: Session) -> GitSource:
         )
     # ``url`` is the table's NOT-NULL location column (phase 38: local
     # rows carry the expanded path there too — git URL shapes and absolute
-    # paths cannot collide).
+    # paths cannot collide). A ``token`` on a local row (phase 121) is
+    # stored inert — local rows are walked, not cloned, so
+    # ``clone_url_for`` never sees it — and, like on git rows, is never
+    # echoed by any output shape.
     return _commit_new(
         GitSource(
             url=path,
@@ -388,6 +419,7 @@ def _create_local_row(payload: GitSourceIn, db: Session) -> GitSource:
             path=path,
             ignore_paths=_validate_ignore_paths(payload.ignore_paths),
             include_hidden=bool(payload.include_hidden),
+            token=payload.token or None,
         ),
         f"a local source with this path already exists: {path}",
         db,
@@ -400,14 +432,26 @@ def patch_git_source(
     payload: GitSourcePatchIn,
     db: Session = Depends(get_db),  # noqa: B008
 ) -> GitSourceOut:
-    """Edit one source's ignore list and/or hidden-folders flag.
+    """Edit one source's ignore list, hidden-folders flag, and/or
+    private-repo token.
 
-    Phase 89 A5 (ignore list) + phase 105 (the flag): 404 unknown
-    id; each PRESENT body field applies independently —
-    ``ignore_paths`` REPLACES the list (normalized + A4-validated,
-    fixed 422 details); ``include_hidden`` sets the flag. Both
-    absent → 200 no-op. Returns the updated row's public shape
-    (id, url, added_at, ignore_paths, include_hidden).
+    Phase 89 A5 (ignore list) + phase 105 (the flag) + phase 121
+    (the token): 404 unknown id; each PRESENT body field applies
+    independently — ``ignore_paths`` REPLACES the list (normalized +
+    A4-validated, fixed 422 details); ``include_hidden`` sets the
+    flag; ``token`` is TRI-STATE (LOCKED A2): absent/None = no change
+    (the row's stored credential survives an edit that does not touch
+    the masked field), non-empty = replace, empty string = clear
+    (stored NULL). A PRESENT token also re-normalizes the (current
+    url, new token) pair with the POST write-path rules — a legacy
+    embedded-token URL gets its userinfo stripped (moved to the
+    column) the first time an explicit credential is written; a clean
+    URL comes back untouched. The 409 backstop: re-normalizing can
+    make the stored URL collide with another row's bare URL (a
+    legacy ``user:pass@`` row and a bare row for the same repo) — the
+    unique index yields the generic 409, never a 500. Returns the
+    updated row's public shape (id, url — masked, added_at,
+    ignore_paths, include_hidden); the token is never echoed.
     """
     row = db.get(GitSource, source_id)
     if row is None:
@@ -416,11 +460,28 @@ def patch_git_source(
         row.ignore_paths = _validate_ignore_paths(payload.ignore_paths)
     if payload.include_hidden is not None:
         row.include_hidden = payload.include_hidden
-    db.commit()
+    if payload.token is not None:
+        # Phase 121 (task 02): the tri-state applies — "" clears
+        # (stored NULL), non-empty replaces. Re-normalize the pair
+        # (see the docstring): a legacy embedded-token URL becomes
+        # bare + column credential.
+        row.url, effective = normalize_credential(row.url, payload.token)
+        row.token = effective or None
+    try:
+        db.commit()
+    except IntegrityError:
+        # The re-normalized URL collided with another row's stored URL
+        # (the legacy-embedded + bare sibling case) — the unique index
+        # is the backstop: a generic 409, never a 500 (the phase-35
+        # convention).
+        db.rollback()
+        raise HTTPException(
+            status_code=409, detail="a git source with this URL already exists"
+        ) from None
     db.refresh(row)
     return GitSourceOut(
         id=row.id,
-        url=row.url,
+        url=sanitize_url(row.url),  # phase 121: the output mask, always
         added_at=row.added_at,
         ignore_paths=row.ignore_paths,
         include_hidden=row.include_hidden,

app/api/sync.py

@@ -27,8 +27,14 @@ decisions):
    URLs) fails loudly (``no sources configured (git or local)``)
    instead of silently importing the legacy local directories;
 3. per resolved row: ``kind=git`` → :func:`scripts.git_sync.clone_or_pull`
-   into ``BOR_SOURCES_DIR/<repo-name>/`` (phase 28 — reused, not
-   re-implemented); ``kind=local`` → the stored directory, re-verified
+   with the phase-121 clone URL (:func:`app.rag.git_sources.clone_url_for`
+   — the row's ``token`` column injected as
+   ``https://x-access-token:<token>@…`` only for https? rows; NULL
+   token → the bare stored URL verbatim, so public repos and legacy
+   embedded-token rows clone exactly as before) into
+   ``BOR_SOURCES_DIR/<repo-name>/`` (phase 28 — reused, not
+   re-implemented; the checkout name stays on the bare URL —
+   credential-free); ``kind=local`` → the stored directory, re-verified
    ``.is_dir()`` **at sync time** (it may have moved/deleted since
    add-time) — a missing directory raises ``local source missing:
    <path>``; a failing clone or a missing local dir aborts before any
@@ -116,7 +122,7 @@ from app.core.auth import require_admin
 from app.core.errors import sanitize_error as _sanitize_error
 from app.db import SessionLocal
 from app.rag.folder_summaries import generate_folder_summaries, missing_folder_summaries
-from app.rag.git_sources import effective_sources
+from app.rag.git_sources import clone_url_for, effective_sources
 from app.rag.importer import ImportSummary, import_sources
 from app.rag.llm import LLMClient, check_models
 from app.rag.overview import regenerate_overview
@@ -293,7 +299,11 @@ async def _run_sync() -> None:
         doc_dates_by_root: dict[str, dict[str, datetime]] = {}
         for row in rows:
             if row.kind == "git":
-                root = clone_or_pull(row.url, sources_root / repo_name(row.url))
+                # Phase 121: the token column is injected into the clone
+                # URL ONLY here (clone_url_for — NULL token → the bare
+                # stored URL verbatim); repo_name stays on the bare URL
+                # so the checkout directory name is credential-free.
+                root = clone_or_pull(clone_url_for(row), sources_root / repo_name(row.url))
                 # Phase 106 (D2): the checkout's per-file last-commit
                 # dates, keyed by the SAME root string the importer
                 # sees (full-history checkouts → true per-file

app/models.py

@@ -286,6 +286,16 @@ class GitSource(Base):
     include_hidden: Mapped[bool] = mapped_column(
         Boolean, default=False, server_default=text("false"), nullable=False
     )
+    #: Private-repo credential (phase 121, LOCKED A2): the PAT the owner
+    #: types into the masked Sources-page field. NULL = public repo (or a
+    #: legacy row whose credential is still embedded in ``url``). Stored
+    #: plaintext BY NECESSITY — the repo must remain cloneable, so the
+    #: raw credential must be recoverable at sync time; the DB is the
+    #: trusted store and is never served to the UI. Injected into the
+    #: clone URL ONLY at clone time
+    #: (:func:`app.rag.git_sources.clone_url_for`); NEVER returned by
+    #: any API shape (the output models gain no token field).
+    token: Mapped[str | None] = mapped_column(Text, default=None)
     added_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), server_default=func.now())
 
 

app/rag/git_sources.py

@@ -21,9 +21,26 @@ CLI: the legacy ``DEFAULT_SOURCES`` fallback).
 (repo URLs of the effective git rows) so existing importers of the old
 name keep working; new code calls :func:`effective_sources` and
 branches on ``row.kind``.
+
+Phase 121 (private git sources) adds the token mechanics next to the
+resolver — three pure helpers, no DB of their own:
+
+* :func:`sanitize_url` — the OUTPUT mask: strips the ``user:pass@``
+  userinfo of ``https?://`` URLs so no API/UI surface ever shows an
+  embedded credential (legacy rows included; the stored value is
+  untouched — LOCKED A2);
+* :func:`clone_url_for` — the CLONE-time credential: a row's
+  ``token`` column is injected into the URL handed to git, and only
+  there (NULL token → the bare stored URL verbatim);
+* :func:`normalize_credential` — the WRITE-path normalizer: an
+  old-style ``https://user:pass@host/repo.git`` URL pasted into the
+  API is stored bare and the embedded credential is moved into the
+  ``token`` column (an explicit ``token`` field wins — LOCKED A6).
 """
 from __future__ import annotations
 
+import logging
+import re
 from typing import Literal
 
 from sqlalchemy import select
@@ -32,6 +49,109 @@ from sqlalchemy.orm import Session
 from app.config import get_settings
 from app.models import GitSource
 
+logger = logging.getLogger(__name__)
+
+#: Phase 121 — the userinfo component of an ``https?://`` URL: the
+#: scheme, a run of one-or-more characters that are neither ``@`` nor
+#: ``/`` (the ``user`` or ``user:pass`` part), and the terminating
+#: ``@``. Deliberately a small anchored regex — never a URL parser
+#: re-serialization: for a credential-free URL there is no match and
+#: the input is returned byte-identical (the phase-50/35 contract that
+#: stored URLs surface verbatim when they carry no credential).
+_USERINFO_RE = re.compile(r"^(https?://)([^/@]+)@")
+
+
+def sanitize_url(url: str) -> str:
+    """Phase 121, LOCKED A2 — the token-free form of a source URL,
+    for API/UI output only.
+
+    Strips the userinfo component of ``https?://`` URLs
+    (``https://user:pass@host/path`` → ``https://host/path``);
+    ``ssh://``, ``git@`` (scp-style), and local paths are left
+    untouched. Idempotent — and byte-identical for URLs that carry no
+    userinfo (no match → the input unchanged, including a ``@`` inside
+    the *path*, which is not userinfo). The stored row value is NOT
+    modified: a legacy row whose credential is still embedded in
+    ``url`` keeps cloning with its original stored URL; this is the
+    output mask that keeps that credential out of every response and
+    the UI (the env-fallback rows get the same treatment — the env
+    *value* itself is untouched, only the response is masked).
+    """
+    return _USERINFO_RE.sub(r"\1", url, count=1)
+
+
+def clone_url_for(row: GitSource) -> str:
+    """Phase 121, LOCKED A2 — the URL git actually clones, with the
+    row's credential injected ONLY here.
+
+    * ``token`` NULL/falsy → ``row.url`` verbatim: public repos and
+      local rows behave byte-identically to pre-phase-121, and a
+      legacy embedded-token row (``token`` NULL, credential in the
+      stored URL) keeps cloning with its ORIGINAL stored URL — the
+      credential keeps working;
+    * an ``https?://`` row with a token →
+      ``https://x-access-token:<token>@<host>/<path>`` — any existing
+      userinfo in the stored URL is replaced by the column credential
+      (``x-access-token`` as the username: GitHub-agnostic, any host
+      that accepts ``https://user:token@`` treats the first component
+      opaquely — the task-02 assumption, task 02 step 4);
+    * a non-https row with a token (``ssh://``/``git@``/local path)
+      → ``row.url`` unchanged + a WARNING log (a token cannot
+      authenticate ssh — the owner must use a deploy key/agent there;
+      the log names the repo via its sanitized URL, never the token).
+
+    ``repo_name`` (and every other checkout-path derivation) keeps
+    operating on the bare ``row.url`` — the checkout directory name is
+    credential-free.
+    """
+    token = row.token
+    if not token:
+        return row.url
+    if not row.url.startswith(("https://", "http://")):
+        logger.warning(
+            "git source %s has a stored token but a non-https? URL — "
+            "a token cannot authenticate ssh/git@ clones; the stored "
+            "URL is used as-is (configure a deploy key or SSH agent "
+            "for private ssh repos)",
+            sanitize_url(row.url),
+        )
+        return row.url
+    bare = sanitize_url(row.url)
+    return bare.replace("://", f"://x-access-token:{token}@", 1)
+
+
+def normalize_credential(url: str, token: str | None) -> tuple[str, str | None]:
+    """Phase 121, LOCKED A6 — the write-path credential normalizer.
+
+    If the (``https?://``-only) URL carries userinfo, it is stripped
+    for storage and the EMBEDDED CREDENTIAL becomes the effective
+    token — UNLESS the caller also sent an explicit ``token``
+    (non-None), which WINS (explicit beats embedded — a blank masked
+    field, i.e. an explicit "", is a deliberate "no credential").
+    Pasting the old-style ``https://user:ghp_…@host/repo.git`` URL
+    still works and lands token-column-clean; the caller stores the
+    bare URL + ``effective_token or None`` (an empty explicit token
+    stores NULL) and runs its duplicate check on the BARE URL, so the
+    same repo with a different token is still the same source
+    (409, not a second row).
+
+    The embedded credential is the *password* part of a
+    ``user:pass`` userinfo (after the first colon — the password may
+    contain further colons), or the whole userinfo run for the
+    username-as-token form (``https://<token>@host/…``, the documented
+    GitHub shape, no colon). Clean URLs and ``ssh://``/``git@``/local
+    paths return ``(url, token)`` untouched — byte-identical
+    pre-phase behavior.
+    """
+    match = _USERINFO_RE.match(url)
+    if match is None:
+        return url, token
+    userinfo = match.group(2)
+    user, sep, password = userinfo.partition(":")
+    embedded = password if sep else userinfo
+    effective = token if token is not None else embedded
+    return sanitize_url(url), effective
+
 
 def effective_sources(db: Session) -> tuple[list[GitSource], Literal["db", "env"]]:
     """``(rows, origin)`` — the effective source rows (both kinds) and

app/schemas.py

@@ -583,6 +583,14 @@ class GitSourceIn(BaseModel):
 
     ``include_hidden`` (phase 105) is optional at create time (absent →
     stored ``False`` — A4).
+
+    ``token`` (phase 121, LOCKED A2) is the masked private-repo
+    credential from the Sources page: optional at create time (absent/
+    None = no credential — public repo), trimmed *before* the length
+    constraints run (the ``_trim_url`` precedent), max 500. It is a
+    WRITE-ONLY field — stored in the dedicated ``git_sources.token``
+    column and NEVER echoed back by any output shape (``GitSourceOut``
+    / ``GitSourceRow`` carry no token field by contract).
     """
 
     kind: Literal["git", "local"] = "git"
@@ -590,6 +598,7 @@ class GitSourceIn(BaseModel):
     path: str | None = Field(default=None, min_length=1, max_length=2000)
     ignore_paths: list[str] | None = Field(default=None)
     include_hidden: bool | None = Field(default=None)
+    token: str | None = Field(default=None, max_length=500)
 
     @field_validator("url", mode="before")
     @classmethod
@@ -601,6 +610,11 @@ class GitSourceIn(BaseModel):
     def _trim_path(cls, v: object) -> object:
         return v.strip() if isinstance(v, str) else v
 
+    @field_validator("token", mode="before")
+    @classmethod
+    def _trim_token(cls, v: object) -> object:
+        return v.strip() if isinstance(v, str) else v
+
 
 class GitSourceOut(BaseModel):
     """One created git source as returned by ``POST`` (phase 35, task 02).
@@ -614,8 +628,18 @@ class GitSourceOut(BaseModel):
     list — non-null (a row created without it reports ``[]``).
     ``include_hidden`` (phase 105) is the stored flag — a row created
     without it reports ``False`` (A4).
+
+    There is deliberately NO ``token`` field (phase 121, LOCKED A2):
+    the private-repo credential is stored in the dedicated
+    ``git_sources.token`` column and is NEVER a response field — it
+    never reaches the UI or any API output. ``extra="forbid"`` makes
+    the omission a structural contract, not an accident: constructing
+    this model with a ``token`` key raises, so a regression that tries
+    to echo the credential back cannot even build the shape.
     """
 
+    model_config = ConfigDict(extra="forbid")
+
     id: uuid.UUID | None
     url: str
     added_at: datetime | None
@@ -637,8 +661,16 @@ class GitSourceRow(BaseModel):
     store a list on) report ``[]``. ``include_hidden`` (phase 105) is
     the row's stored flag — env-fallback rows (no DB row to store a flag
     on) report ``False`` (the ``ignore_paths: []`` precedent).
+
+    There is deliberately NO ``token`` field (phase 121, LOCKED A2):
+    same contract as :class:`GitSourceOut` — the credential never
+    reaches the UI or any API output, and ``extra="forbid"`` makes the
+    omission structural (constructing a row with a ``token`` key
+    raises).
     """
 
+    model_config = ConfigDict(extra="forbid")
+
     id: uuid.UUID | None
     kind: Literal["git", "local"]
     url: str
@@ -658,12 +690,24 @@ class GitSourcePatchIn(BaseModel):
     normalized + A4-validated, becomes the row's whole list — empty
     list clears all; every pre-phase-105 client always sends the
     list, so their behavior is byte-identical). ``include_hidden``
-    (phase 105) when present sets the stored flag. Both absent →
-    200 no-op (the row is untouched).
+    (phase 105) when present sets the stored flag. ``token`` (phase
+    121, LOCKED A2) is TRI-STATE — the three-way semantics the masked
+    edit field depends on: **absent/None = no change** (keep the row's
+    stored credential), **non-empty = replace**, **empty string =
+    clear** (the UI offers replace; clear exists for API completeness).
+    Trimmed *before* the length constraints run (the ``GitSourceIn``
+    ``_trim_token`` precedent — whitespace-only counts as a clear),
+    max 500. All absent → 200 no-op (the row is untouched).
     """
 
     ignore_paths: list[str] | None = Field(default=None)
     include_hidden: bool | None = Field(default=None)
+    token: str | None = Field(default=None, max_length=500)
+
+    @field_validator("token", mode="before")
+    @classmethod
+    def _trim_token(cls, v: object) -> object:
+        return v.strip() if isinstance(v, str) else v
 
 
 class GitSourceList(BaseModel):