phase: 121_git_source_tokens
Build and Push Containers / build-and-push-app (push) Successful in 2m3s
Build and Push Containers / build-and-push-db (push) Failing after 14s

**Phase 121 final verification pass — all green** (all 4 tasks already in `complete/`; verified, no defects found, no changes needed)

- Verified implementation vs phase design: migration `0021` (reversible, round-tripped via `alembic downgrade base` + `upgrade head` → head `0021`), `GitSource.token` column, `normalize_credential`/`clone_url_for`/`sanitize_url`, clone callers switched (`sync.py`, `import_docs.py`), masked token fields in add form + editor, `extra="forbid"` output shapes
- Tests: `uv run pytest` → 2662 passed, 0 failed (exit 0); `uv run pytest --cov=app --cov-report=term-missing` → TOTAL **99%** (≥90% gate)
- Lint/types: `uv run ruff check .` → All checks passed; `uv run pyright` → 0 errors, 0 warnings
- E2E in isolation: `uv run pytest tests/e2e/test_git_source_tokens.py -v --no-cov` → **4 passed**

Completion criteria:
1. Private repo (UI add or pasted embedded-token URL) clones with injected token; token absent from every API response, page text, title attr, and full HTML — **PASS** (integration raw-JSON assertions + E2E `_assert_token_nowhere`)
2. Legacy embedded-token rows still clone from stored URL; output sanitized — **PASS** (`test_sync_legacy_row_clones_with_original_stored_url`, `test_get_masks_legacy_embedded_token_row`, env-fallback masking)
3. Public/local sources byte-identical — **PASS** (verbatim-URL + no-userinfo-unchanged tests)
4. pytest / coverage / ruff / pyright — **PASS** (see above)
5. Commit + phase move — harness responsibility; task files already in `complete/`, changes left in working tree (no commit made, per protocol)

Notable: no deviations; DB left at head, functional. Next pending phase: **122_image_documents** (then 123_chat_image_questions).
This commit is contained in:
2026-09-24 20:51:39 -04:00
parent 3a0fc3db05
commit 0f77e9a876
35 changed files with 2894 additions and 48 deletions
@@ -0,0 +1,48 @@
# Phase 121 — Private git sources: a token that never reaches the UI or the API
**Source:** `TODO.md` L5 — "Need a way to add private repos without exposing the token in the UI (like when adding an https repo `https://myuser:ghp_xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx@github.com/myuser/my-private-repo.git`)"
**Story:** n/a (feature request; extends the phase-28/35/38 git/local sources and phase-89 per-source settings assets).
**Context:** `app/models.py:245` — `GitSource`: `url: Text UNIQUE NOT NULL`, `kind` ("git"/"local"), `path`, `ignore_paths` (JSONB), `include_hidden`, `added_at` — no token column. `app/schemas.py` — `GitSourceIn` (L561: `kind`, `url` min 1/max 500, `path`, `ignore_paths`, `include_hidden`; `_trim_url` before-validator L594), `GitSourceOut` (L605: `url: str`), `GitSourceRow` (L626: `url`), `GitSourcePatchIn` (L651). `app/api/git_sources.py` — `URL_RE = ^(?:https?://|ssh://|git@)` (L195; a prefix match, so `user:token@` URLs pass); POST validates the prefix (L344–345) and duplicates via `GitSource.url == url` (L349); GET list / GET single return `row.url` RAW (L235, L250, L297, L423) — an embedded token is echoed to any browser (the leak); the local-source upload endpoint (L432). `app/api/sync.py:296` — `clone_or_pull(row.url, sources_root / repo_name(row.url))`; `scripts/git_sync.py` — `clone_or_pull`, `repo_name`; `scripts/import_docs.py::_resolve_sources` — the CLI's second clone caller (both consume `app.rag.git_sources.effective_sources`, L27). `frontend/assets/git-sources.js` — add form `#git-source-url` (L241; submit `body: (url) => ({ url })` L945); every display site renders `s.url` raw (list cell L395–406 incl. the `title` attr, delete row L533, edit modal L697, ignore-list context L763/L830). `alembic/` — migrations.
## Objective
Private repos are added with a bare URL plus an optional MASKED token field. The token lives in a dedicated DB column, is injected only into the clone URL at sync/clone time, and is absent from every API response and UI surface — including legacy rows that already embed the token in `url` (those are sanitized on output but keep working).
## Dependencies
- `120_failed_turn_retry` (todo) — pipeline predecessor (execution order) only; no code dependency.
- Code dependencies (all complete): phase 35/38 `GitSource` kinds + `effective_sources`, phase 89 per-source settings (the PATCH-field precedent), phase 28 `clone_or_pull`/`repo_name`.
## Design (shared by all tasks — the executor reads this, not the chat)
- **Storage (task 01, LOCKED A2):** `GitSource.token` — `Text NULL` (NULL = public/no credential; **plaintext by necessity** — the repo must remain cloneable, so the raw credential must be recoverable; the Postgres DB is the trusted store and is never served to the UI; there is deliberately no external secrets backend). `GitSourceIn.token: str | None = None` (max 500, trimmed); `GitSourcePatchIn.token: str | None = None` — PATCH semantics: **absent/None = no change, non-empty = replace, empty string = clear** (the UI offers replace; clear exists for API completeness). `GitSourceOut`/`GitSourceRow` gain NO token field — no response shape ever carries it (LOCKED A2).
- **Normalization (task 02):** on POST (and PATCH when a new url/token arrives), the server normalizes: if the incoming URL contains userinfo (`user:pass@host`, only for `https?://` URLs — ssh/`git@` carry no userinfo), the **userinfo is stripped** for storage and the embedded credential is moved into `token` — UNLESS the caller also sent an explicit `token` field, which WINS (explicit beats embedded). Pasting the old-style `https://user:ghp_…@github.com/x/y.git` URL still works and ends up token-column-clean. The duplicate check (L349) runs on the NORMALIZED bare URL, so the same repo with a different token is still the same source (409, not a second row).
- **Clone-time credential (task 02):** `clone_url_for(row) -> str` in `app/rag/git_sources.py` (next to `effective_sources`): `row.url` unchanged when `token` is NULL; otherwise inject `https://x-access-token:<token>@<host>/<path>` (https rows only — a token on a non-https row is a no-op with a warning log). `repo_name` keeps operating on the bare `row.url`. Callers switch from `row.url` to `clone_url_for(row)`: `app/api/sync.py:296` and `scripts/import_docs.py::_resolve_sources` (both already import from `app.rag.git_sources`).
- **Output sanitization (task 02, LOCKED A2):** every API surface that returns a git URL runs it through `sanitize_url(url)` (new, in `app/rag/git_sources.py`): strips the userinfo component (`https://…@host/…` → `https://host/…`), leaves ssh/`git@`/local paths untouched. Applied to `GitSourceOut.url` / `GitSourceRow.url` construction (GET list L235/L297, GET single, the `BOR_GIT_SOURCES` env fallback rows L250 — env rows can embed tokens too) and to any sync-status field echoing a repo URL (grep for `url=` in the sync responses). Belt-and-braces for legacy embedded-token rows whose credential is NOT in the `token` column: their DB value is untouched (the clone still authenticates from the stored URL) but no API/UI output ever shows the credential.
- **UI (task 03):** the add form gains a second field — a masked `<input type="password" id="git-source-token">`, optional, labelled "Token (private repos)" with a visible "optional" hint; submit sends `{ url, token }` (token omitted when blank). The edit modal mirrors it with placeholder "leave blank to keep the current token" (blank → omit from PATCH = no change). Every display site keeps rendering `s.url` — now bare by server sanitization, so list cells, `title` attributes, the delete row, and the ignore-list context become token-free with no per-site change. No new CSS beyond reusing the existing form-field styles (the theme's input treatment).
- **NOT touched:** local-kind sources (no URL credential), the `BOR_GIT_SOURCES` env parsing (its rows are sanitized on OUTPUT only), the upload endpoint, sync scheduling, and the Sources page layout.
## Tasks
1. `01_token_storage.md` — migration + `GitSource.token` + input schemas (`GitSourceIn`/`GitSourcePatchIn`); no token in any output shape.
2. `02_clone_url_and_sanitization.md` — URL/token normalization on write, `clone_url_for` at clone time, `sanitize_url` on every output.
3. `03_ui_token_field.md` — masked token field in the add form + edit modal; display stays `s.url` (now bare).
4. `04_token_tests.md` — unit + integration + isolated E2E `test_git_source_tokens.py`.
## Testing & Quality
- Unit: `tests/unit/test_git_source_token.py` (new, task 04) — `sanitize_url` (https userinfo stripped, ssh/git@/local untouched, no-userinfo unchanged), `clone_url_for` (NULL token → bare URL; token → injected; non-https token → bare + no crash), normalization (embedded token moved to the column when no explicit token; explicit token wins; duplicate on bare URL).
- Integration: `tests/integration/test_git_sources_api.py` (extend) — POST with `token` → GET list/single responses contain the token NOWHERE (assert on the raw JSON text) and show the bare URL; POST with an old-style embedded-token URL → stored bare + token column populated, responses clean; PATCH token replace/clear semantics; the sync flow builds the clone URL with the injected token (mock `clone_or_pull`).
- E2E: `tests/e2e/test_git_source_tokens.py` (new, task 04) — isolated run per AGENTS.md §4: add a private repo through the Sources UI (bare URL + token) → the list row shows the bare URL, the token is absent from the page text, the `title` attribute, and `GET /api/git-sources` JSON; edit the row (blank token) → no 4xx, token kept.
- Coverage: **>90%** on `app/` (validate.sh gate).
## Completion Criteria
- [ ] A private repo added via the UI (or a pasted embedded-token URL) syncs/clones fine, and its token appears in NO API response, NO page text, and NO attribute.
- [ ] Legacy embedded-token rows (pre-phase) still clone, and their API/UI output is token-free.
- [ ] Public repos and local sources behave byte-identically to before.
- [ ] `uv run pytest` green; `uv run pytest --cov=app --cov-report=term-missing` TOTAL >90%; `uv run ruff check . && uv run pyright` clean.
- [ ] One `--no-gpg-sign` commit; phase dir moved to `.agents/phases/complete/` by the pipeline gate.
## Locked decisions
- **A2 — the token is stored PLAINTEXT in a dedicated `GitSource.token` column (cloneability requires the raw credential; no external secrets backend), is NEVER returned by any API shape, and legacy embedded-token URLs are sanitized on output while keeping their stored value for clones (owner-confirmed 2026-09-24, roadmap confirmation).**
- **A6 — pasting an old-style embedded-token URL is accepted and normalized (userinfo → `token` column); an explicit `token` field wins over an embedded one (owner-confirmed: same confirmation — the proposed design).**
## Commit
```bash
git add app/ alembic/ frontend/ scripts/ tests/ .agents/phases/ && git commit --no-gpg-sign -m "feat(sources): add private git repos with a masked token that never reaches the UI or API"
```
@@ -0,0 +1,38 @@
# Task 01 — Token storage: model, migration, input schemas
**Phase:** `121_git_source_tokens` · **Source:** `TODO.md:5` — "Need a way to add private repos without exposing the token in the UI (like when adding an https repo `https://myuser:ghp_xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx@github.com/myuser/my-private-repo.git`)"
## Objective
The `git_sources` table can hold a per-row token, and the add/patch input shapes can carry one — while NO output shape (`GitSourceOut`, `GitSourceRow`, list, single, env-fallback rows) ever can.
## Work
1. `app/models.py` — `GitSource` (L245): add
```python
#: Private-repo credential (phase 121, LOCKED A2): the PAT the owner
#: types into the masked Sources-page field. NULL = public repo (or a
#: legacy row whose credential is still embedded in ``url``). Stored
#: plaintext BY NECESSITY — the repo must remain cloneable, so the
#: raw credential must be recoverable at sync time; the DB is the
#: trusted store and is never served to the UI. Injected into the
#: clone URL ONLY at clone time
#: (:func:`app.rag.git_sources.clone_url_for`); NEVER returned by
#: any API shape (the output models gain no token field).
token: Mapped[str | None] = mapped_column(Text, default=None)
```
(docstring first, then the column — the phase-89/105 field-docstring house style).
2. `alembic/versions/` — new revision (head of the current chain): `op.add_column("git_sources", sa.Column("token", sa.Text(), nullable=True))` + downgrade `op.drop_column`. Follow the existing migration file conventions (check the latest revision for the revision/down_revision pattern).
3. `app/schemas.py`:
- `GitSourceIn` (L561): add `token: str | None = Field(default=None, max_length=500)` with a `before`-mode trim validator (the `_trim_url` L594 precedent) — plus the docstring note: masked input from the Sources page; absent/None = no credential.
- `GitSourcePatchIn` (L651): add `token: str | None = Field(default=None, max_length=500)` — docstring the PATCH tri-state: **absent/None = no change, non-empty = replace, empty string = clear**.
- `GitSourceOut` (L605) / `GitSourceRow` (L626): add NO field; extend their docstrings with the explicit "no token — never a response field (phase 121)" note so the omission is a documented contract, not an accident.
4. No endpoint changes in this task (acceptance of `token` and normalization are task 02) — the extra field on the input models is inert until then (Pydantic would currently just pass it through unused; task 02 consumes it).
## Testing & Quality
- Unit: `tests/unit/test_git_source_token.py` (task 04 extends): `GitSourceIn`/`GitSourcePatchIn` accept/trim `token`; `GitSourceOut`/`GitSourceRow` reject a `token` key (they are output models built from rows — assert constructing them with a token kwarg raises).
- Integration: `tests/integration/test_git_sources_api.py` (task 04) — after `alembic upgrade head`, `git_sources.token` exists (a `SELECT` sanity check in the existing test fixtures).
- Coverage: **>90%** on `app/` for the touched modules.
## Completion Criteria
- [ ] `uv run alembic upgrade head` applies the new revision on a clean DB and the downgrade removes the column.
- [ ] The ORM round-trips a `token` value; output models have no token field (grep + unit assertion).
- [ ] `uv run pytest` green; `uv run ruff check . && uv run pyright` clean.
@@ -0,0 +1,29 @@
# Task 02 — Clone-time credential + output sanitization
**Phase:** `121_git_source_tokens` · **Source:** `TODO.md:5` — "Need a way to add private repos without exposing the token in the UI …"
## Objective
The token is injected into the clone URL only at clone time, old-style embedded-token URLs are normalized into the column on write, and every URL that leaves the API is token-free — including legacy rows and env-fallback rows.
## Work
1. `app/rag/git_sources.py` — add two pure helpers (unit-testable, no DB):
- `sanitize_url(url: str) -> str` — strip the userinfo component of `https?://` URLs (`https://user:pass@host/path` → `https://host/path`); leave `ssh://`, `git@`, and local paths untouched; idempotent. Use a small regex (`^(https?://)([^/@]+)@` → `\1`) — never a URL parser that re-serializes (byte-identical output for clean URLs is a requirement: the phase-50/35 contract is that stored URLs surface verbatim when they carry no credential).
- `clone_url_for(row) -> str` — `row.url` when `row.token` is falsy; for an `https://` row with a token, inject `https://x-access-token:<token>@<host>/<path>` (replace any existing userinfo with the column credential); for a non-https row with a token, log a warning and return `row.url` unchanged (a token cannot authenticate ssh — the owner must use a deploy key/agent there).
- Also `normalize_credential(url, token) -> (bare_url, effective_token)` — the write-path normalizer: if `url` (https only) contains userinfo, strip it → bare URL, and the embedded credential becomes `effective_token` UNLESS `token` is non-None (explicit wins, LOCKED A6). Returns the input untouched for clean URLs.
2. `app/api/git_sources.py`:
- POST git source (L342–355): run `normalize_credential(payload.url, payload.token)`; store the BARE url + `effective_token`; the duplicate check (L349) runs on the bare URL.
- PATCH (the url/token branch): when `payload.url` or `payload.token` is present, re-normalize the (current or new) pair with the same rules — PATCH token tri-state from task 01 (None = no change, `""` = clear → store NULL, non-empty = replace).
- Every output construction runs `sanitize_url` on the URL before it enters the response: the DB-row list path (L235/L297), the GET single path (L423), and the `BOR_GIT_SOURCES` env-fallback rows (L250 — an env URL can embed a token; the ENV VALUE itself is untouched, only the response is masked). Grep the router for any other `url=` response field (including sync-status echoes — `app/api/sync.py` responses that surface a repo URL get the same treatment) and sanitize those too.
3. `app/api/sync.py` (L296) + `scripts/import_docs.py::_resolve_sources` — swap `row.url` → `clone_url_for(row)` at the clone call site (`clone_or_pull(clone_url_for(row), sources_root / repo_name(row.url))` — `repo_name` stays on the bare URL so the checkout directory name is credential-free).
4. ASSUMPTION: `x-access-token` as the injected userinfo username (GitHub-agnostic — any git host that accepts `https://user:token@` treats the first component opaquely; `oauth2:` is also common, but `x-access-token` works on GitHub and GitLab and reads as non-identifying).
## Testing & Quality
- Unit: `tests/unit/test_git_source_token.py` (task 04 finalizes) — `sanitize_url` (strip/no-op/idempotent/ssh/git@/local), `clone_url_for` (NULL token; token injected; non-https token no-op), `normalize_credential` (embedded→column, explicit wins, clean URL untouched).
- Integration: `tests/integration/test_git_sources_api.py` (task 04) — POST embedded-token URL → row.url bare + row.token populated; GET list JSON (raw text) contains the token NOWHERE; sync with a token row (mock `clone_or_pull`) receives the injected URL and a credential-free checkout path.
- Coverage: **>90%** on the touched modules.
## Completion Criteria
- [ ] No token string in ANY API response for a token-bearing row (integration assertion on raw JSON text).
- [ ] A legacy row (token embedded in the stored `url`, `token` NULL) still produces the ORIGINAL stored URL at clone time (the credential keeps working) but its API output is masked.
- [ ] `repo_name` / checkout paths are credential-free.
- [ ] `uv run pytest` green; `uv run ruff check . && uv run pyright` clean.
@@ -0,0 +1,30 @@
# Task 03 — UI: masked token field on add + edit
**Phase:** `121_git_source_tokens` · **Source:** `TODO.md:5` — "Need a way to add private repos without exposing the token in the UI …"
## Objective
The Sources-page git-source form takes a separate masked token field (add and edit); every display surface shows the bare URL (server-sanitized) and the token is nowhere in the DOM.
## Work
1. `frontend/assets/git-sources.js`:
- Add form (the `#git-source-url` field at L241, submit wiring at L942–945): add a second labelled field
```html
<label for="git-source-token">Token <span class="field-hint">optional — private repos</span></label>
<input type="password" id="git-source-token" autocomplete="off" placeholder="ghp_… or another PAT">
```
(reuse the existing form-field markup/CSS classes from the url field — the theme's input treatment, no new CSS needed beyond the existing `.field-hint` or an equivalent inline span). Submit body becomes `(url, token) => ({ url, ...(token ? { token } : {}) })` — blank token = key omitted (None = no credential).
- Edit modal (the url display/edit at L697): the same masked token field, placeholder "leave blank to keep the current token"; PATCH body includes `token` ONLY when non-blank (blank → omitted → no change — the task-01 tri-state).
- Display sites (L395–406 list cell incl. the `title` attribute, L533 delete row, L763/L830 ignore-list context): keep rendering `s.url` UNCHANGED — the server now returns bare URLs, so nothing to do per site. Add a source-comment note (one line) that URLs arrive sanitized server-side (phase 121) and the UI must never re-embed a credential.
2. `frontend/assets/styles.css` — only if the "optional" hint span has no existing class to reuse: a minimal `.field-hint` (muted color, contrast ≥4.5:1 per PLAN §7, small).
3. ASSUMPTION: the password field is `type="password"` with `autocomplete="off"` (a PAT is not a site credential; browsers must not offer to save it).
## Testing & Quality
- Unit: `tests/unit/test_git_source_token.py` (task 04) — house-style source assertions: `#git-source-token` is `type="password"` and `autocomplete="off"`; the submit body omits a blank token; the edit PATCH omits a blank token; no display site concatenates a token.
- E2E: `tests/e2e/test_git_source_tokens.py` (task 04) — the UI scenarios.
- Coverage: n/a (frontend) — the validate.sh `app/` gate must stay green.
## Completion Criteria
- [ ] Adding a private repo through the UI with a token succeeds; the list row shows the bare URL.
- [ ] The token is absent from the rendered page text, the `title` attribute, and the Sources-page DOM (E2E assertion).
- [ ] Editing with a blank token keeps the existing credential (integration: the PATCH tri-state).
- [ ] `uv run pytest` green; `uv run ruff check . && uv run pyright` clean.
@@ -0,0 +1,36 @@
# Task 04 — Token tests: unit + integration + isolated E2E
**Phase:** `121_git_source_tokens` · **Source:** `TODO.md:5` — "Need a way to add private repos without exposing the token in the UI …"
## Objective
Pin the whole credential contract: helpers are pure and correct, no token ever crosses the API boundary (raw-JSON assertion), legacy rows stay cloneable and masked, and the UI never renders a credential.
## Work
1. `tests/unit/test_git_source_token.py` (new):
- `sanitize_url` — https userinfo stripped (`https://myuser:ghp_x@github.com/x/y.git` → `https://github.com/x/y.git`), clean https unchanged byte-identically, `ssh://git@host/x.git` untouched, `git@github.com:x/y.git` untouched, a local path untouched, idempotent on already-clean URLs.
- `clone_url_for` — `token` NULL → `row.url` verbatim; https + token → `https://x-access-token:<token>@host/path` (existing userinfo REPLACED); non-https + token → `row.url` (no crash, warning logged).
- `normalize_credential` — embedded userinfo → bare URL + token populated; explicit token wins over embedded; clean URL + None token → unchanged.
- Output models: `GitSourceOut`/`GitSourceRow` reject a `token` kwarg (no response field can ever carry it).
- Frontend source assertions (house style): `#git-source-token` is `type="password"` + `autocomplete="off"`; submit/PATCH omit a blank token.
2. `tests/integration/test_git_sources_api.py` (extend):
- POST `{url: "https://github.com/acme/private.git", token: "ghp_test123"}` → 201; `GET /api/git-sources` raw response TEXT does not contain `ghp_test123`; the row's `url` is the bare URL; `GET` single likewise.
- POST an old-style `https://myuser:ghp_legacy@github.com/acme/legacy.git` (no token field) → stored `url` bare, `token` = `ghp_legacy`; responses token-free.
- POST the same repo a second time (different token) → 409 (duplicate on the bare URL).
- PATCH token tri-state: absent → kept; non-empty → replaced (clone URL uses the new one); `""` → cleared (clone URL bare again).
- Legacy-row simulation (insert a row directly with the embedded URL, `token` NULL): `GET` output masked; the sync path (mock `clone_or_pull`) still receives the ORIGINAL stored URL (clone works).
- Sync flow: a token row → `clone_or_pull` called with the injected URL; the checkout path is credential-free.
3. `tests/e2e/test_git_source_tokens.py` (new — isolated run per AGENTS.md §4: `uv run pytest tests/e2e/test_git_source_tokens.py -v --no-cov`):
- Open the Sources page (admin), add a git source with a bare URL + a distinctive fake token (`ghp_e2esecret…`);
- assert: the list row renders the BARE URL; the token string is absent from `document.body.innerText`, from every `title` attribute, and from the `GET /api/git-sources` JSON (via a `page.request.get` inside the test);
- open the edit modal: the token field is blank (never pre-filled — a password must not be echoed back, so it is simply empty by design); save with it blank → 200, row intact;
- remove the source (cleanup) — the list is empty again.
4. Run the full gate: `uv run pytest`, `uv run pytest --cov=app --cov-report=term-missing` (TOTAL >90%), the isolated E2E file, `uv run ruff check . && uv run pyright`.
## Testing & Quality
- This task IS the phase's test suite (see Work).
- Coverage: **>90%** on `app/` — the phase's `app/` code (helpers + router + schema + model) is fully exercised by the unit/integration cases.
## Completion Criteria
- [ ] All test artifacts exist and pass; the isolated E2E file passes standalone.
- [ ] The raw-JSON "token nowhere" assertion covers list AND single AND sync-status surfaces.
- [ ] `uv run pytest --cov=app` TOTAL >90%; lint + types clean.