Experimental/brain-of-reese
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

phase: 121_git_source_tokens
Build and Push Containers / build-and-push-app (push) Successful in 2m3s
Details
Build and Push Containers / build-and-push-db (push) Failing after 14s
Details

Browse Source 
**Phase 121 final verification pass — all green** (all 4 tasks already in `complete/`; verified, no defects found, no changes needed)

- Verified implementation vs phase design: migration `0021` (reversible, round-tripped via `alembic downgrade base` + `upgrade head` → head `0021`), `GitSource.token` column, `normalize_credential`/`clone_url_for`/`sanitize_url`, clone callers switched (`sync.py`, `import_docs.py`), masked token fields in add form + editor, `extra="forbid"` output shapes
- Tests: `uv run pytest` → 2662 passed, 0 failed (exit 0); `uv run pytest --cov=app --cov-report=term-missing` → TOTAL **99%** (≥90% gate)
- Lint/types: `uv run ruff check .` → All checks passed; `uv run pyright` → 0 errors, 0 warnings
- E2E in isolation: `uv run pytest tests/e2e/test_git_source_tokens.py -v --no-cov` → **4 passed**

Completion criteria:
1. Private repo (UI add or pasted embedded-token URL) clones with injected token; token absent from every API response, page text, title attr, and full HTML — **PASS** (integration raw-JSON assertions + E2E `_assert_token_nowhere`)
2. Legacy embedded-token rows still clone from stored URL; output sanitized — **PASS** (`test_sync_legacy_row_clones_with_original_stored_url`, `test_get_masks_legacy_embedded_token_row`, env-fallback masking)
3. Public/local sources byte-identical — **PASS** (verbatim-URL + no-userinfo-unchanged tests)
4. pytest / coverage / ruff / pyright — **PASS** (see above)
5. Commit + phase move — harness responsibility; task files already in `complete/`, changes left in working tree (no commit made, per protocol)

Notable: no deviations; DB left at head, functional. Next pending phase: **122_image_documents** (then 123_chat_image_questions).
This commit is contained in:
ducoterra
2026-09-24 20:51:39 -04:00
parent 3a0fc3db05
commit 0f77e9a876
35 changed files with 2894 additions and 48 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.agents/phases/todo/121_git_source_tokens/00_phase.md → .agents/phases/complete/121_git_source_tokens/00_phase.md
View File
.agents/phases/todo/121_git_source_tokens/01_token_storage.md → .agents/phases/complete/121_git_source_tokens/01_token_storage.md
View File
.agents/phases/todo/121_git_source_tokens/02_clone_url_and_sanitization.md → .agents/phases/complete/121_git_source_tokens/02_clone_url_and_sanitization.md
View File
.agents/phases/todo/121_git_source_tokens/03_ui_token_field.md → .agents/phases/complete/121_git_source_tokens/03_ui_token_field.md
View File
.agents/phases/todo/121_git_source_tokens/04_token_tests.md → .agents/phases/complete/121_git_source_tokens/04_token_tests.md
View File
.agents/reports/121_git_source_tokens/121_git_source_tokens__00_phase.a1.err
View File
.agents/reports/121_git_source_tokens/121_git_source_tokens__00_phase.a1.md
+15
View File
@@ -0,0 +1,15 @@
**Phase 121 final verification pass — all green** (all 4 tasks already in `complete/`; verified, no defects found, no changes needed)
- Verified implementation vs phase design: migration `0021` (reversible, round-tripped via `alembic downgrade base` + `upgrade head` → head `0021`), `GitSource.token` column, `normalize_credential`/`clone_url_for`/`sanitize_url`, clone callers switched (`sync.py`, `import_docs.py`), masked token fields in add form + editor, `extra="forbid"` output shapes
- Tests: `uv run pytest` → 2662 passed, 0 failed (exit 0); `uv run pytest --cov=app --cov-report=term-missing` → TOTAL **99%** (≥90% gate)
- Lint/types: `uv run ruff check .` → All checks passed; `uv run pyright` → 0 errors, 0 warnings
- E2E in isolation: `uv run pytest tests/e2e/test_git_source_tokens.py -v --no-cov` → **4 passed**
Completion criteria:
1. Private repo (UI add or pasted embedded-token URL) clones with injected token; token absent from every API response, page text, title attr, and full HTML — **PASS** (integration raw-JSON assertions + E2E `_assert_token_nowhere`)
2. Legacy embedded-token rows still clone from stored URL; output sanitized — **PASS** (`test_sync_legacy_row_clones_with_original_stored_url`, `test_get_masks_legacy_embedded_token_row`, env-fallback masking)
3. Public/local sources byte-identical — **PASS** (verbatim-URL + no-userinfo-unchanged tests)
4. pytest / coverage / ruff / pyright — **PASS** (see above)
5. Commit + phase move — harness responsibility; task files already in `complete/`, changes left in working tree (no commit made, per protocol)
Notable: no deviations; DB left at head, functional. Next pending phase: **122_image_documents** (then 123_chat_image_questions).
.agents/reports/121_git_source_tokens/121_git_source_tokens__00_phase.a1.validate
+105
View File
@@ -0,0 +1,105 @@
........................................................................ [ 2%]
........................................................................ [ 5%]
........................................................................ [ 8%]
........................................................................ [ 10%]
........................................................................ [ 13%]
........................................................................ [ 16%]
........................................................................ [ 19%]
........................................................................ [ 21%]
........................................................................ [ 24%]
........................................................................ [ 27%]
........................................................................ [ 29%]
........................................................................ [ 32%]
........................................................................ [ 35%]
........................................................................ [ 38%]
........................................................................ [ 40%]
........................................................................ [ 43%]
........................................................................ [ 46%]
........................................................................ [ 48%]
........................................................................ [ 51%]
........................................................................ [ 54%]
........................................................................ [ 57%]
........................................................................ [ 59%]
........................................................................ [ 62%]
........................................................................ [ 65%]
........................................................................ [ 67%]
........................................................................ [ 70%]
........................................................................ [ 73%]
........................................................................ [ 76%]
........................................................................ [ 78%]
........................................................................ [ 81%]
........................................................................ [ 84%]
........................................................................ [ 86%]
........................................................................ [ 89%]
........................................................................ [ 92%]
........................................................................ [ 95%]
........................................................................ [ 97%]
.......................................................... [100%]
=============================== warnings summary ===============================
.venv/lib/python3.13/site-packages/fastapi/testclient.py:1
/var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
from starlette.testclient import TestClient as TestClient # noqa
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
================================ tests coverage ================================
_______________ coverage: platform linux, python 3.13.13-final-0 _______________
Name Stmts Miss Cover
--------------------------------------------------
app/__init__.py 1 0 100%
app/api/__init__.py 0 0 100%
app/api/auth.py 52 0 100%
app/api/chat.py 226 1 99%
app/api/chats.py 110 0 100%
app/api/config.py 13 0 100%
app/api/doc_drafts.py 99 0 100%
app/api/docs.py 156 1 99%
app/api/git_sources.py 241 0 100%
app/api/health.py 10 0 100%
app/api/steering.py 42 0 100%
app/api/suggestions.py 33 0 100%
app/api/sync.py 139 0 100%
app/api/tokens.py 40 0 100%
app/api/ui_settings.py 55 0 100%
app/config.py 231 0 100%
app/core/__init__.py 0 0 100%
app/core/auth.py 45 0 100%
app/core/caching.py 124 0 100%
app/core/debugging.py 29 2 93%
app/core/docs_push.py 39 0 100%
app/core/errors.py 5 0 100%
app/core/logging.py 13 0 100%
app/core/rate_limit.py 44 0 100%
app/core/security_headers.py 20 0 100%
app/core/theming.py 38 0 100%
app/core/tokens.py 44 0 100%
app/db.py 22 0 100%
app/main.py 66 0 100%
app/models.py 129 0 100%
app/rag/__init__.py 0 0 100%
app/rag/agent.py 347 1 99%
app/rag/archive_upload.py 134 0 100%
app/rag/chunker.py 206 4 98%
app/rag/doc_dates.py 18 0 100%
app/rag/folder_summaries.py 123 0 100%
app/rag/git_sources.py 38 0 100%
app/rag/importer.py 219 3 99%
app/rag/llm.py 244 1 99%
app/rag/overview.py 71 0 100%
app/rag/prompts.py 102 0 100%
app/rag/retriever.py 242 3 99%
app/rag/scaffolding.py 55 0 100%
app/rag/source_removal.py 41 0 100%
app/rag/sources_meta.py 16 0 100%
app/rag/suggestions.py 27 0 100%
app/rag/summarizer.py 24 0 100%
app/schemas.py 344 0 100%
--------------------------------------------------
TOTAL 4317 16 99%
coverage gate: app/ 99% (>90%) OK
All checks passed!
0 errors, 0 warnings, 0 informations
WARNING: there is a new pyright version available (v1.1.411 -> v1.1.414).
Please install the new version or set PYRIGHT_PYTHON_FORCE_VERSION to `latest`
validation OK
.agents/reports/121_git_source_tokens/121_git_source_tokens__01_token_storage.a1.err
View File
.agents/reports/121_git_source_tokens/121_git_source_tokens__01_token_storage.a1.md
+19
View File
@@ -0,0 +1,19 @@
Task 01 complete. Report:
**Implemented**
- `GitSource.token` — `Text NULL` column in `app/models.py` with the phase-121/LOCKED-A2 field docstring (NULL = public/legacy row)
- `alembic/versions/0021_git_source_token.py` — additive, fully reversible (drop_column downgrade); verified upgrade/downgrade/upgrade on the live dev DB
- `app/schemas.py`: `GitSourceIn.token` + `GitSourcePatchIn.token` (max 500, before-mode trim validators, tri-state patch docstring); `GitSourceOut`/`GitSourceRow` gain **no** field — docstring contract + `extra="forbid"` so any `token` on construction raises
- Tests: `tests/unit/test_git_source_token.py` (21 tests) + `tests/integration/test_migration_0021.py` (4 tests, house A13 pattern incl. ORM round-trip)
**Results**
- `uv run pytest` — 2602 passed
- `uv run pytest --cov=app --cov-report=term-missing` — TOTAL **99%** (>90% ✓)
- `uv run ruff check . && uv run pyright` — clean (0 errors; the two deliberate `token=` rejection lines carry house-style `# type: ignore[reportCallIssue]`)
- `uv run alembic upgrade head` / `downgrade 0020` — apply/reverse cleanly; dev DB left at head (0021)
**Decisions**
- Added the before-mode trim to `GitSourcePatchIn.token` as well (house `_trim_url` precedent; whitespace-only = clear, consistent with the tri-state)
- `extra="forbid"` on both output shapes makes "no token ever" structural (ChatMessage precedent), satisfying the task's "construction with a token kwarg raises" pin
**Next pending task:** `02_clone_url_and_sanitization.md` (normalization on write, `clone_url_for`, `sanitize_url`)
.agents/reports/121_git_source_tokens/121_git_source_tokens__01_token_storage.a1.validate
+105
View File
@@ -0,0 +1,105 @@
........................................................................ [ 2%]
........................................................................ [ 5%]
........................................................................ [ 8%]
........................................................................ [ 11%]
........................................................................ [ 13%]
........................................................................ [ 16%]
........................................................................ [ 19%]
........................................................................ [ 22%]
........................................................................ [ 24%]
........................................................................ [ 27%]
........................................................................ [ 30%]
........................................................................ [ 33%]
........................................................................ [ 35%]
........................................................................ [ 38%]
........................................................................ [ 41%]
........................................................................ [ 44%]
........................................................................ [ 47%]
........................................................................ [ 49%]
........................................................................ [ 52%]
........................................................................ [ 55%]
........................................................................ [ 58%]
........................................................................ [ 60%]
........................................................................ [ 63%]
........................................................................ [ 66%]
........................................................................ [ 69%]
........................................................................ [ 71%]
........................................................................ [ 74%]
........................................................................ [ 77%]
........................................................................ [ 80%]
........................................................................ [ 83%]
........................................................................ [ 85%]
........................................................................ [ 88%]
........................................................................ [ 91%]
........................................................................ [ 94%]
........................................................................ [ 96%]
........................................................................ [ 99%]
.......... [100%]
=============================== warnings summary ===============================
.venv/lib/python3.13/site-packages/fastapi/testclient.py:1
/var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
from starlette.testclient import TestClient as TestClient # noqa
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
================================ tests coverage ================================
_______________ coverage: platform linux, python 3.13.13-final-0 _______________
Name Stmts Miss Cover
--------------------------------------------------
app/__init__.py 1 0 100%
app/api/__init__.py 0 0 100%
app/api/auth.py 52 0 100%
app/api/chat.py 226 1 99%
app/api/chats.py 110 0 100%
app/api/config.py 13 0 100%
app/api/doc_drafts.py 99 0 100%
app/api/docs.py 156 1 99%
app/api/git_sources.py 232 0 100%
app/api/health.py 10 0 100%
app/api/steering.py 42 0 100%
app/api/suggestions.py 33 0 100%
app/api/sync.py 139 0 100%
app/api/tokens.py 40 0 100%
app/api/ui_settings.py 55 0 100%
app/config.py 231 0 100%
app/core/__init__.py 0 0 100%
app/core/auth.py 45 0 100%
app/core/caching.py 124 0 100%
app/core/debugging.py 29 2 93%
app/core/docs_push.py 39 0 100%
app/core/errors.py 5 0 100%
app/core/logging.py 13 0 100%
app/core/rate_limit.py 44 0 100%
app/core/security_headers.py 20 0 100%
app/core/theming.py 38 0 100%
app/core/tokens.py 44 0 100%
app/db.py 22 0 100%
app/main.py 66 0 100%
app/models.py 129 0 100%
app/rag/__init__.py 0 0 100%
app/rag/agent.py 347 1 99%
app/rag/archive_upload.py 134 0 100%
app/rag/chunker.py 206 4 98%
app/rag/doc_dates.py 18 0 100%
app/rag/folder_summaries.py 123 0 100%
app/rag/git_sources.py 14 0 100%
app/rag/importer.py 219 3 99%
app/rag/llm.py 244 1 99%
app/rag/overview.py 71 0 100%
app/rag/prompts.py 102 0 100%
app/rag/retriever.py 242 3 99%
app/rag/scaffolding.py 55 0 100%
app/rag/source_removal.py 41 0 100%
app/rag/sources_meta.py 16 0 100%
app/rag/suggestions.py 27 0 100%
app/rag/summarizer.py 24 0 100%
app/schemas.py 344 0 100%
--------------------------------------------------
TOTAL 4284 16 99%
coverage gate: app/ 99% (>90%) OK
All checks passed!
0 errors, 0 warnings, 0 informations
WARNING: there is a new pyright version available (v1.1.411 -> v1.1.414).
Please install the new version or set PYRIGHT_PYTHON_FORCE_VERSION to `latest`
validation OK
.agents/reports/121_git_source_tokens/121_git_source_tokens__02_clone_url_and_sanitization.a1.err
View File
.agents/reports/121_git_source_tokens/121_git_source_tokens__02_clone_url_and_sanitization.a1.md
+17
View File
@@ -0,0 +1,17 @@
All gates green. Task 02 complete.
**Implemented**
- `app/rag/git_sources.py`: three pure helpers — `sanitize_url` (anchored-regex userinfo strip, byte-identical/idempotent for clean URLs), `clone_url_for` (NULL token → stored URL verbatim; https? + token → `https://x-access-token:<token>@…`; non-https + token → unchanged + warning), `normalize_credential` (write-path: embedded userinfo → bare URL + token column; explicit token wins, LOCKED A6)
- `app/api/git_sources.py`: POST normalizes (embedded `user:pass@` → bare URL + column; duplicate check on bare URL → 409); PATCH token tri-state (absent/None = no change, non-empty = replace, `""` = clear) with legacy-row re-normalization + IntegrityError→generic-409 backstop; `sanitize_url` on all 4 output sites (list DB rows, env rows, POST 201, PATCH 200)
- `app/api/sync.py` + `scripts/import_docs.py`: clone call sites now use `clone_url_for(row)`; `repo_name` stays on the bare URL (credential-free checkout paths)
**Tests / gates**
- 32 new unit tests (`tests/unit/test_git_source_token.py`), 13 new integration tests (`tests/integration/test_git_sources_api.py`) — token absent from raw JSON text of every response, legacy rows clone with original stored URL, sync receives injected URL
- `uv run pytest --cov=app --cov-report=term-missing` → exit 0, TOTAL **99%** (>90%; touched modules 100%)
- `uv run ruff check .` clean; `uv run pyright` 0 errors; sanity: `tests/e2e/test_git_sources_admin.py` 6/6 in isolation
**Notable decisions**
- The "embedded credential" moved to the column is the *password* part of `user:pass@` (whole run for the no-colon `https://<token>@host` form) — only that authenticates via the injected `x-access-token:<password>@`
- A token on a `kind=local` row is stored inert (never cloned, never echoed); env-fallback URLs are masked on output only
**Next pending task:** `03_ui_token_field.md`
.agents/reports/121_git_source_tokens/121_git_source_tokens__02_clone_url_and_sanitization.a1.validate
+105
View File
@@ -0,0 +1,105 @@
........................................................................ [ 2%]
........................................................................ [ 5%]
........................................................................ [ 8%]
........................................................................ [ 10%]
........................................................................ [ 13%]
........................................................................ [ 16%]
........................................................................ [ 19%]
........................................................................ [ 21%]
........................................................................ [ 24%]
........................................................................ [ 27%]
........................................................................ [ 29%]
........................................................................ [ 32%]
........................................................................ [ 35%]
........................................................................ [ 38%]
........................................................................ [ 40%]
........................................................................ [ 43%]
........................................................................ [ 46%]
........................................................................ [ 49%]
........................................................................ [ 51%]
........................................................................ [ 54%]
........................................................................ [ 57%]
........................................................................ [ 59%]
........................................................................ [ 62%]
........................................................................ [ 65%]
........................................................................ [ 68%]
........................................................................ [ 70%]
........................................................................ [ 73%]
........................................................................ [ 76%]
........................................................................ [ 79%]
........................................................................ [ 81%]
........................................................................ [ 84%]
........................................................................ [ 87%]
........................................................................ [ 89%]
........................................................................ [ 92%]
........................................................................ [ 95%]
........................................................................ [ 98%]
................................................... [100%]
=============================== warnings summary ===============================
.venv/lib/python3.13/site-packages/fastapi/testclient.py:1
/var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
from starlette.testclient import TestClient as TestClient # noqa
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
================================ tests coverage ================================
_______________ coverage: platform linux, python 3.13.13-final-0 _______________
Name Stmts Miss Cover
--------------------------------------------------
app/__init__.py 1 0 100%
app/api/__init__.py 0 0 100%
app/api/auth.py 52 0 100%
app/api/chat.py 226 1 99%
app/api/chats.py 110 0 100%
app/api/config.py 13 0 100%
app/api/doc_drafts.py 99 0 100%
app/api/docs.py 156 1 99%
app/api/git_sources.py 241 0 100%
app/api/health.py 10 0 100%
app/api/steering.py 42 0 100%
app/api/suggestions.py 33 0 100%
app/api/sync.py 139 0 100%
app/api/tokens.py 40 0 100%
app/api/ui_settings.py 55 0 100%
app/config.py 231 0 100%
app/core/__init__.py 0 0 100%
app/core/auth.py 45 0 100%
app/core/caching.py 124 0 100%
app/core/debugging.py 29 2 93%
app/core/docs_push.py 39 0 100%
app/core/errors.py 5 0 100%
app/core/logging.py 13 0 100%
app/core/rate_limit.py 44 0 100%
app/core/security_headers.py 20 0 100%
app/core/theming.py 38 0 100%
app/core/tokens.py 44 0 100%
app/db.py 22 0 100%
app/main.py 66 0 100%
app/models.py 129 0 100%
app/rag/__init__.py 0 0 100%
app/rag/agent.py 347 1 99%
app/rag/archive_upload.py 134 0 100%
app/rag/chunker.py 206 4 98%
app/rag/doc_dates.py 18 0 100%
app/rag/folder_summaries.py 123 0 100%
app/rag/git_sources.py 38 0 100%
app/rag/importer.py 219 3 99%
app/rag/llm.py 244 1 99%
app/rag/overview.py 71 0 100%
app/rag/prompts.py 102 0 100%
app/rag/retriever.py 242 3 99%
app/rag/scaffolding.py 55 0 100%
app/rag/source_removal.py 41 0 100%
app/rag/sources_meta.py 16 0 100%
app/rag/suggestions.py 27 0 100%
app/rag/summarizer.py 24 0 100%
app/schemas.py 344 0 100%
--------------------------------------------------
TOTAL 4317 16 99%
coverage gate: app/ 99% (>90%) OK
All checks passed!
0 errors, 0 warnings, 0 informations
WARNING: there is a new pyright version available (v1.1.411 -> v1.1.414).
Please install the new version or set PYRIGHT_PYTHON_FORCE_VERSION to `latest`
validation OK
.agents/reports/121_git_source_tokens/121_git_source_tokens__03_ui_token_field.a1.err
View File
.agents/reports/121_git_source_tokens/121_git_source_tokens__03_ui_token_field.a1.md
+17
View File
@@ -0,0 +1,17 @@
All checks pass and the diff is clean. Final verification summary:
- `uv run pytest --cov=app --cov-report=term-missing` → **2650 passed**, TOTAL coverage **99%** (>90% gate)
- `uv run ruff check .` → clean; `uv run pyright` → **0 errors, 0 warnings**
- Affected story E2Es in isolation: `test_source_ignore_paths.py` 6 passed, `test_git_sources_admin.py` 6 passed, `test_hidden_folders_toggle.py` 6 passed
**Implemented (task 03 — UI masked token field):**
- **`frontend/index.html`**: add form gains optional masked `#git-source-token` (`type="password"`, `autocomplete="off"`, "optional — private repos" hint span); the per-row editor dialog mirrors it with `#ignore-editor-token` ("leave blank to keep the current token"); dialog title widened "Ignored files and folders" → "Source settings" to match its new scope
- **`frontend/assets/git-sources.js`**: submit body `(url, token) => ({ url, ...(token ? { token } : {}) })` (blank = key omitted); editor token always opens/resets blank, PATCH body includes `token` only when non-blank (tri-state no-change); display sites keep rendering server-sanitized `s.url` with the required one-line note; phase-121 docstring bullet
- **`frontend/assets/styles.css`**: `#git-source-token` grouped with the URL input (incl. mobile rule), minimal `.field-hint` (ink-soft, 5.1:1 AA), `.ignore-editor-token` full-width 44px box
- **Tests**: 7 new house-style source pins in `tests/unit/test_git_source_token.py` (masking, blank-omission on POST+PATCH, display sites, docstring, CSS); updated `test_source_ignore_paths.py` save-body pin + id guard for the intentional dialog extension
**Notable decisions:**
- The task's "edit modal (L697)" is not a distinct modal — verified against the phase-authoring commit, L697 is the per-row "Ignore paths" editor dialog (the only per-row edit surface); extended that, matching the phase's "edit row (blank token) → token kept" contract and the "Sources page layout untouched" boundary
- Task-03 unit pins landed now (task file defers to 04, but they assert this task's changes); the phase E2E file stays task 04's deliverable
**Next pending task:** `.agents/phases/todo/121_git_source_tokens/04_token_tests.md` (finalize unit/integration + isolated E2E `test_git_source_tokens.py`)
.agents/reports/121_git_source_tokens/121_git_source_tokens__03_ui_token_field.a1.validate
+105
View File
@@ -0,0 +1,105 @@
........................................................................ [ 2%]
........................................................................ [ 5%]
........................................................................ [ 8%]
........................................................................ [ 10%]
........................................................................ [ 13%]
........................................................................ [ 16%]
........................................................................ [ 19%]
........................................................................ [ 21%]
........................................................................ [ 24%]
........................................................................ [ 27%]
........................................................................ [ 29%]
........................................................................ [ 32%]
........................................................................ [ 35%]
........................................................................ [ 38%]
........................................................................ [ 40%]
........................................................................ [ 43%]
........................................................................ [ 46%]
........................................................................ [ 48%]
........................................................................ [ 51%]
........................................................................ [ 54%]
........................................................................ [ 57%]
........................................................................ [ 59%]
........................................................................ [ 62%]
........................................................................ [ 65%]
........................................................................ [ 67%]
........................................................................ [ 70%]
........................................................................ [ 73%]
........................................................................ [ 76%]
........................................................................ [ 78%]
........................................................................ [ 81%]
........................................................................ [ 84%]
........................................................................ [ 86%]
........................................................................ [ 89%]
........................................................................ [ 92%]
........................................................................ [ 95%]
........................................................................ [ 97%]
.......................................................... [100%]
=============================== warnings summary ===============================
.venv/lib/python3.13/site-packages/fastapi/testclient.py:1
/var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
from starlette.testclient import TestClient as TestClient # noqa
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
================================ tests coverage ================================
_______________ coverage: platform linux, python 3.13.13-final-0 _______________
Name Stmts Miss Cover
--------------------------------------------------
app/__init__.py 1 0 100%
app/api/__init__.py 0 0 100%
app/api/auth.py 52 0 100%
app/api/chat.py 226 1 99%
app/api/chats.py 110 0 100%
app/api/config.py 13 0 100%
app/api/doc_drafts.py 99 0 100%
app/api/docs.py 156 1 99%
app/api/git_sources.py 241 0 100%
app/api/health.py 10 0 100%
app/api/steering.py 42 0 100%
app/api/suggestions.py 33 0 100%
app/api/sync.py 139 0 100%
app/api/tokens.py 40 0 100%
app/api/ui_settings.py 55 0 100%
app/config.py 231 0 100%
app/core/__init__.py 0 0 100%
app/core/auth.py 45 0 100%
app/core/caching.py 124 0 100%
app/core/debugging.py 29 2 93%
app/core/docs_push.py 39 0 100%
app/core/errors.py 5 0 100%
app/core/logging.py 13 0 100%
app/core/rate_limit.py 44 0 100%
app/core/security_headers.py 20 0 100%
app/core/theming.py 38 0 100%
app/core/tokens.py 44 0 100%
app/db.py 22 0 100%
app/main.py 66 0 100%
app/models.py 129 0 100%
app/rag/__init__.py 0 0 100%
app/rag/agent.py 347 1 99%
app/rag/archive_upload.py 134 0 100%
app/rag/chunker.py 206 4 98%
app/rag/doc_dates.py 18 0 100%
app/rag/folder_summaries.py 123 0 100%
app/rag/git_sources.py 38 0 100%
app/rag/importer.py 219 3 99%
app/rag/llm.py 244 1 99%
app/rag/overview.py 71 0 100%
app/rag/prompts.py 102 0 100%
app/rag/retriever.py 242 3 99%
app/rag/scaffolding.py 55 0 100%
app/rag/source_removal.py 41 0 100%
app/rag/sources_meta.py 16 0 100%
app/rag/suggestions.py 27 0 100%
app/rag/summarizer.py 24 0 100%
app/schemas.py 344 0 100%
--------------------------------------------------
TOTAL 4317 16 99%
coverage gate: app/ 99% (>90%) OK
All checks passed!
0 errors, 0 warnings, 0 informations
WARNING: there is a new pyright version available (v1.1.411 -> v1.1.414).
Please install the new version or set PYRIGHT_PYTHON_FORCE_VERSION to `latest`
validation OK
.agents/reports/121_git_source_tokens/121_git_source_tokens__04_token_tests.a1.err
View File
.agents/reports/121_git_source_tokens/121_git_source_tokens__04_token_tests.a1.md
+15
View File
@@ -0,0 +1,15 @@
All work is verified and the tree is as it should be (only my new E2E file added; tasks 01–03 changes already in the working tree, untouched by me, no commits).
**Phase 121 / Task 04 — Token tests: complete**
Implemented:
- `tests/e2e/test_git_source_tokens.py` (new, 4 tests, isolated port 8146, `BOR_GIT_SOURCES=""` + tmp sources/upload dirs pinned): add via UI (bare URL + masked token) → row renders bare URL in cell + `title` attr, token absent from body text / every `title` / page HTML / raw `GET /api/git-sources` JSON, DB = bare URL + token column; old-style `user:token@` paste normalized (A6) and hidden; editor token field `type=password`, opens blank, blank save → single PATCH 200 omitting the `token` key, stored token kept; confirm-modal removal → 204, list empty.
- Verified task-01–03 deliverables already in place and green: `tests/unit/test_git_source_token.py` (50 tests — sanitize/clone/normalize/output-model/frontend pins) and the `tests/integration/test_git_sources_api.py` token block (POST/409/PATCH tri-state/legacy mask/sync-clone raw-JSON "token nowhere" on list, 201, PATCH, sync-status surfaces).
Gates (all green):
- `uv run pytest` → 2650 passed
- `uv run pytest --cov=app --cov-report=term-missing` → TOTAL **99%** (>90%); `app/api/git_sources.py`, `app/rag/git_sources.py`, `app/schemas.py`, `app/models.py` all 100%
- `uv run pytest tests/e2e/test_git_source_tokens.py -v --no-cov` → 4 passed
- `uv run ruff check . && uv run pyright` → clean, 0 errors
Decisions: no defects found in prior tasks; only fix was in my new file (Playwright `APIResponse.text` is a method — `r.text()`; body text via `page.evaluate` per house style). Next pending task: none in this phase — all 4 tasks done, phase is complete.
.agents/reports/121_git_source_tokens/121_git_source_tokens__04_token_tests.a1.validate
+105
View File
@@ -0,0 +1,105 @@
........................................................................ [ 2%]
........................................................................ [ 5%]
........................................................................ [ 8%]
........................................................................ [ 10%]
........................................................................ [ 13%]
........................................................................ [ 16%]
........................................................................ [ 19%]
........................................................................ [ 21%]
........................................................................ [ 24%]
........................................................................ [ 27%]
........................................................................ [ 29%]
........................................................................ [ 32%]
........................................................................ [ 35%]
........................................................................ [ 38%]
........................................................................ [ 40%]
........................................................................ [ 43%]
........................................................................ [ 46%]
........................................................................ [ 48%]
........................................................................ [ 51%]
........................................................................ [ 54%]
........................................................................ [ 57%]
........................................................................ [ 59%]
........................................................................ [ 62%]
........................................................................ [ 65%]
........................................................................ [ 67%]
........................................................................ [ 70%]
........................................................................ [ 73%]
........................................................................ [ 76%]
........................................................................ [ 78%]
........................................................................ [ 81%]
........................................................................ [ 84%]
........................................................................ [ 86%]
........................................................................ [ 89%]
........................................................................ [ 92%]
........................................................................ [ 95%]
........................................................................ [ 97%]
.......................................................... [100%]
=============================== warnings summary ===============================
.venv/lib/python3.13/site-packages/fastapi/testclient.py:1
/var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead.
from starlette.testclient import TestClient as TestClient # noqa
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
================================ tests coverage ================================
_______________ coverage: platform linux, python 3.13.13-final-0 _______________
Name Stmts Miss Cover
--------------------------------------------------
app/__init__.py 1 0 100%
app/api/__init__.py 0 0 100%
app/api/auth.py 52 0 100%
app/api/chat.py 226 1 99%
app/api/chats.py 110 0 100%
app/api/config.py 13 0 100%
app/api/doc_drafts.py 99 0 100%
app/api/docs.py 156 1 99%
app/api/git_sources.py 241 0 100%
app/api/health.py 10 0 100%
app/api/steering.py 42 0 100%
app/api/suggestions.py 33 0 100%
app/api/sync.py 139 0 100%
app/api/tokens.py 40 0 100%
app/api/ui_settings.py 55 0 100%
app/config.py 231 0 100%
app/core/__init__.py 0 0 100%
app/core/auth.py 45 0 100%
app/core/caching.py 124 0 100%
app/core/debugging.py 29 2 93%
app/core/docs_push.py 39 0 100%
app/core/errors.py 5 0 100%
app/core/logging.py 13 0 100%
app/core/rate_limit.py 44 0 100%
app/core/security_headers.py 20 0 100%
app/core/theming.py 38 0 100%
app/core/tokens.py 44 0 100%
app/db.py 22 0 100%
app/main.py 66 0 100%
app/models.py 129 0 100%
app/rag/__init__.py 0 0 100%
app/rag/agent.py 347 1 99%
app/rag/archive_upload.py 134 0 100%
app/rag/chunker.py 206 4 98%
app/rag/doc_dates.py 18 0 100%
app/rag/folder_summaries.py 123 0 100%
app/rag/git_sources.py 38 0 100%
app/rag/importer.py 219 3 99%
app/rag/llm.py 244 1 99%
app/rag/overview.py 71 0 100%
app/rag/prompts.py 102 0 100%
app/rag/retriever.py 242 3 99%
app/rag/scaffolding.py 55 0 100%
app/rag/source_removal.py 41 0 100%
app/rag/sources_meta.py 16 0 100%
app/rag/suggestions.py 27 0 100%
app/rag/summarizer.py 24 0 100%
app/schemas.py 344 0 100%
--------------------------------------------------
TOTAL 4317 16 99%
coverage gate: app/ 99% (>90%) OK
All checks passed!
0 errors, 0 warnings, 0 informations
WARNING: there is a new pyright version available (v1.1.411 -> v1.1.414).
Please install the new version or set PYRIGHT_PYTHON_FORCE_VERSION to `latest`
validation OK
alembic/versions/0021_git_source_token.py
+47
View File
@@ -0,0 +1,47 @@
"""git_sources.token: private-repo credential column (phase 121)
Revision ID: 0021
Revises: 0020
Create Date: 2026-09-24
Phase 121 (private git sources: a token that never reaches the UI or
the API — task 01, storage only):
* ``git_sources.token`` — TEXT NULLABLE, no server default: the private
repo credential (LOCKED A2) the owner types into the masked
Sources-page field. NULL = public repo (or a legacy row whose
credential is still embedded in ``url`` — those rows keep their
stored value, which is what authenticates the clone, and are
sanitized on OUTPUT only, task 02). Stored plaintext BY NECESSITY:
the repo must remain cloneable, so the raw credential must be
recoverable at sync time; the Postgres DB is the trusted store and is
never served to the UI. The column is injected into the clone URL
ONLY at clone time (task 02's ``clone_url_for``) and is NEVER
returned by any API shape (the output models gain no token field —
the omission is a documented contract).
One additive, fully reversible migration (A13); no other schema
change. Normalization of embedded-token URLs on write and output
sanitization are code (tasks 02/03) — this revision only carries the
column.
"""
from __future__ import annotations
import sqlalchemy as sa
from alembic import op
revision = "0021"
down_revision = "0020"
branch_labels = None
depends_on = None
def upgrade() -> None:
op.add_column("git_sources", sa.Column("token", sa.Text(), nullable=True))
def downgrade() -> None:
# The token column is the only 0021 artefact — dropping it leaves
# 0020's schema byte-identical (A13, fully reversible).
op.drop_column("git_sources", "token")
app/api/git_sources.py
+78 -17
View File
@@ -49,11 +49,16 @@ embeddings) committed first, then the app-managed on-disk dir). The
whole router sits behind :func:`app.core.auth.require_admin` —
anonymous callers get 403 on every route.
No credential-echo path: git URLs may embed ``user:pass@`` (phase 32's
masking discipline), so every git 409/422 detail is a fixed generic
string that never repeats the submitted URL. Local paths are not
secrets — the local 422/409 details name the (expanded) path so the
owner sees exactly which directory failed.
No credential-echo path (phase 32's masking discipline, extended by
phase 121 — LOCKED A2): git URLs may embed ``user:pass@``, so (a)
every git 409/422 detail is a fixed generic string that never repeats
the submitted URL, and (b) every URL that LEAVES the API is masked
through :func:`app.rag.git_sources.sanitize_url` before it enters a
response (DB rows, env-fallback rows, POST 201, PATCH 200) — a legacy
row whose credential is still embedded in the stored ``url`` clones
fine (the stored value is untouched) but its API/UI output is
bare. Local paths are not secrets — the local 422/409 details name
the (expanded) path so the owner sees exactly which directory failed.
Scope boundary (phase locked decisions): the CRUD routes do NOT
clone or import anything — the existing Sync button performs that, and
@@ -110,6 +115,7 @@ from app.rag.archive_upload import (
swap_in,
unpack_archive,
)
from app.rag.git_sources import normalize_credential, sanitize_url
from app.rag.importer import normalize_ignore_path
from app.rag.llm import LLMClient
from app.rag.overview import regenerate_overview
@@ -220,6 +226,11 @@ def list_git_sources(
git-only) with null ``id``/``added_at``, ``ignore_paths: []`` and
``include_hidden: False`` (no DB row to store a list or a flag on),
and ``from_env: true``.
Every URL is masked on the way out (phase 121, LOCKED A2 —
:func:`sanitize_url`): an env value or a legacy stored URL may
embed ``user:pass@`` — the env value and the DB value are
untouched, only the response is bare.
"""
rows = db.scalars(
select(GitSource).order_by(GitSource.added_at.asc(), GitSource.id.asc())
@@ -232,7 +243,7 @@ def list_git_sources(
GitSourceRow(
id=row.id,
kind=cast(Literal["git", "local"], row.kind),
url=row.url,
url=sanitize_url(row.url), # phase 121: never echo userinfo
path=row.path,
added_at=row.added_at,
ignore_paths=row.ignore_paths or [],
@@ -247,7 +258,7 @@ def list_git_sources(
GitSourceRow(
id=None,
kind="git",
url=url,
url=sanitize_url(url), # phase 121: an env URL can embed a token
path=None,
added_at=None,
ignore_paths=[],
@@ -290,11 +301,20 @@ def create_git_source(
``include_hidden`` (phase 105) — optional, both kinds: absent →
stored ``False`` (A4), present → stored as sent; the stored flag is
what is reported.
``token`` (phase 121, LOCKED A2) — the masked private-repo
credential: write-only, stored in the dedicated column, never
echoed (the response has no token field by contract). Git rows
are normalized on the way in (``normalize_credential``): an
old-style embedded ``user:pass@`` URL is stored bare with the
credential in the token column, an explicit ``token`` wins over
the embedded one (LOCKED A6), and the duplicate check runs on the
bare URL.
"""
row = _create_git_row(payload, db) if payload.kind == "git" else _create_local_row(payload, db)
return GitSourceOut(
id=row.id,
url=row.url,
url=sanitize_url(row.url), # phase 121: the output mask, always
added_at=row.added_at,
ignore_paths=row.ignore_paths,
include_hidden=row.include_hidden,
@@ -346,6 +366,13 @@ def _create_git_row(payload: GitSourceIn, db: Session) -> GitSource:
raise HTTPException(
status_code=422, detail="not a valid git URL (expected https://, ssh:// or git@…)"
)
# Phase 121 (task 02, LOCKED A6): normalize the credential — an
# old-style embedded ``user:pass@`` URL is stored BARE and the
# embedded credential moves to the token column; an explicit
# ``token`` field wins over the embedded one. The duplicate check
# below runs on the BARE URL, so the same repo pasted with a
# different credential is the same source (409, not a second row).
url, effective_token = normalize_credential(url, payload.token)
if db.scalar(select(GitSource).where(GitSource.url == url)) is not None:
raise HTTPException(status_code=409, detail="a git source with this URL already exists")
return _commit_new(
@@ -354,6 +381,7 @@ def _create_git_row(payload: GitSourceIn, db: Session) -> GitSource:
kind="git",
ignore_paths=_validate_ignore_paths(payload.ignore_paths),
include_hidden=bool(payload.include_hidden),
token=effective_token or None,
),
"a git source with this URL already exists",
db,
@@ -380,7 +408,10 @@ def _create_local_row(payload: GitSourceIn, db: Session) -> GitSource:
)
# ``url`` is the table's NOT-NULL location column (phase 38: local
# rows carry the expanded path there too — git URL shapes and absolute
# paths cannot collide).
# paths cannot collide). A ``token`` on a local row (phase 121) is
# stored inert — local rows are walked, not cloned, so
# ``clone_url_for`` never sees it — and, like on git rows, is never
# echoed by any output shape.
return _commit_new(
GitSource(
url=path,
@@ -388,6 +419,7 @@ def _create_local_row(payload: GitSourceIn, db: Session) -> GitSource:
path=path,
ignore_paths=_validate_ignore_paths(payload.ignore_paths),
include_hidden=bool(payload.include_hidden),
token=payload.token or None,
),
f"a local source with this path already exists: {path}",
db,
@@ -400,14 +432,26 @@ def patch_git_source(
payload: GitSourcePatchIn,
db: Session = Depends(get_db), # noqa: B008
) -> GitSourceOut:
"""Edit one source's ignore list and/or hidden-folders flag.
"""Edit one source's ignore list, hidden-folders flag, and/or
private-repo token.
Phase 89 A5 (ignore list) + phase 105 (the flag): 404 unknown
id; each PRESENT body field applies independently —
``ignore_paths`` REPLACES the list (normalized + A4-validated,
fixed 422 details); ``include_hidden`` sets the flag. Both
absent → 200 no-op. Returns the updated row's public shape
(id, url, added_at, ignore_paths, include_hidden).
Phase 89 A5 (ignore list) + phase 105 (the flag) + phase 121
(the token): 404 unknown id; each PRESENT body field applies
independently — ``ignore_paths`` REPLACES the list (normalized +
A4-validated, fixed 422 details); ``include_hidden`` sets the
flag; ``token`` is TRI-STATE (LOCKED A2): absent/None = no change
(the row's stored credential survives an edit that does not touch
the masked field), non-empty = replace, empty string = clear
(stored NULL). A PRESENT token also re-normalizes the (current
url, new token) pair with the POST write-path rules — a legacy
embedded-token URL gets its userinfo stripped (moved to the
column) the first time an explicit credential is written; a clean
URL comes back untouched. The 409 backstop: re-normalizing can
make the stored URL collide with another row's bare URL (a
legacy ``user:pass@`` row and a bare row for the same repo) — the
unique index yields the generic 409, never a 500. Returns the
updated row's public shape (id, url — masked, added_at,
ignore_paths, include_hidden); the token is never echoed.
"""
row = db.get(GitSource, source_id)
if row is None:
@@ -416,11 +460,28 @@ def patch_git_source(
row.ignore_paths = _validate_ignore_paths(payload.ignore_paths)
if payload.include_hidden is not None:
row.include_hidden = payload.include_hidden
if payload.token is not None:
# Phase 121 (task 02): the tri-state applies — "" clears
# (stored NULL), non-empty replaces. Re-normalize the pair
# (see the docstring): a legacy embedded-token URL becomes
# bare + column credential.
row.url, effective = normalize_credential(row.url, payload.token)
row.token = effective or None
try:
db.commit()
except IntegrityError:
# The re-normalized URL collided with another row's stored URL
# (the legacy-embedded + bare sibling case) — the unique index
# is the backstop: a generic 409, never a 500 (the phase-35
# convention).
db.rollback()
raise HTTPException(
status_code=409, detail="a git source with this URL already exists"
) from None
db.refresh(row)
return GitSourceOut(
id=row.id,
url=row.url,
url=sanitize_url(row.url), # phase 121: the output mask, always
added_at=row.added_at,
ignore_paths=row.ignore_paths,
include_hidden=row.include_hidden,
app/api/sync.py
+14 -4
View File
@@ -27,8 +27,14 @@ decisions):
URLs) fails loudly (``no sources configured (git or local)``)
instead of silently importing the legacy local directories;
3. per resolved row: ``kind=git`` → :func:`scripts.git_sync.clone_or_pull`
into ``BOR_SOURCES_DIR/<repo-name>/`` (phase 28 — reused, not
re-implemented); ``kind=local`` → the stored directory, re-verified
with the phase-121 clone URL (:func:`app.rag.git_sources.clone_url_for`
— the row's ``token`` column injected as
``https://x-access-token:<token>@…`` only for https? rows; NULL
token → the bare stored URL verbatim, so public repos and legacy
embedded-token rows clone exactly as before) into
``BOR_SOURCES_DIR/<repo-name>/`` (phase 28 — reused, not
re-implemented; the checkout name stays on the bare URL —
credential-free); ``kind=local`` → the stored directory, re-verified
``.is_dir()`` **at sync time** (it may have moved/deleted since
add-time) — a missing directory raises ``local source missing:
<path>``; a failing clone or a missing local dir aborts before any
@@ -116,7 +122,7 @@ from app.core.auth import require_admin
from app.core.errors import sanitize_error as _sanitize_error
from app.db import SessionLocal
from app.rag.folder_summaries import generate_folder_summaries, missing_folder_summaries
from app.rag.git_sources import effective_sources
from app.rag.git_sources import clone_url_for, effective_sources
from app.rag.importer import ImportSummary, import_sources
from app.rag.llm import LLMClient, check_models
from app.rag.overview import regenerate_overview
@@ -293,7 +299,11 @@ async def _run_sync() -> None:
doc_dates_by_root: dict[str, dict[str, datetime]] = {}
for row in rows:
if row.kind == "git":
root = clone_or_pull(row.url, sources_root / repo_name(row.url))
# Phase 121: the token column is injected into the clone
# URL ONLY here (clone_url_for — NULL token → the bare
# stored URL verbatim); repo_name stays on the bare URL
# so the checkout directory name is credential-free.
root = clone_or_pull(clone_url_for(row), sources_root / repo_name(row.url))
# Phase 106 (D2): the checkout's per-file last-commit
# dates, keyed by the SAME root string the importer
# sees (full-history checkouts → true per-file
app/models.py
+10
View File
@@ -286,6 +286,16 @@ class GitSource(Base):
include_hidden: Mapped[bool] = mapped_column(
Boolean, default=False, server_default=text("false"), nullable=False
)
#: Private-repo credential (phase 121, LOCKED A2): the PAT the owner
#: types into the masked Sources-page field. NULL = public repo (or a
#: legacy row whose credential is still embedded in ``url``). Stored
#: plaintext BY NECESSITY — the repo must remain cloneable, so the
#: raw credential must be recoverable at sync time; the DB is the
#: trusted store and is never served to the UI. Injected into the
#: clone URL ONLY at clone time
#: (:func:`app.rag.git_sources.clone_url_for`); NEVER returned by
#: any API shape (the output models gain no token field).
token: Mapped[str | None] = mapped_column(Text, default=None)
added_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), server_default=func.now())
app/rag/git_sources.py
+120
View File
@@ -21,9 +21,26 @@ CLI: the legacy ``DEFAULT_SOURCES`` fallback).
(repo URLs of the effective git rows) so existing importers of the old
name keep working; new code calls :func:`effective_sources` and
branches on ``row.kind``.
Phase 121 (private git sources) adds the token mechanics next to the
resolver — three pure helpers, no DB of their own:
* :func:`sanitize_url` — the OUTPUT mask: strips the ``user:pass@``
userinfo of ``https?://`` URLs so no API/UI surface ever shows an
embedded credential (legacy rows included; the stored value is
untouched — LOCKED A2);
* :func:`clone_url_for` — the CLONE-time credential: a row's
``token`` column is injected into the URL handed to git, and only
there (NULL token → the bare stored URL verbatim);
* :func:`normalize_credential` — the WRITE-path normalizer: an
old-style ``https://user:pass@host/repo.git`` URL pasted into the
API is stored bare and the embedded credential is moved into the
``token`` column (an explicit ``token`` field wins — LOCKED A6).
"""
from __future__ import annotations
import logging
import re
from typing import Literal
from sqlalchemy import select
@@ -32,6 +49,109 @@ from sqlalchemy.orm import Session
from app.config import get_settings
from app.models import GitSource
logger = logging.getLogger(__name__)
#: Phase 121 — the userinfo component of an ``https?://`` URL: the
#: scheme, a run of one-or-more characters that are neither ``@`` nor
#: ``/`` (the ``user`` or ``user:pass`` part), and the terminating
#: ``@``. Deliberately a small anchored regex — never a URL parser
#: re-serialization: for a credential-free URL there is no match and
#: the input is returned byte-identical (the phase-50/35 contract that
#: stored URLs surface verbatim when they carry no credential).
_USERINFO_RE = re.compile(r"^(https?://)([^/@]+)@")
def sanitize_url(url: str) -> str:
"""Phase 121, LOCKED A2 — the token-free form of a source URL,
for API/UI output only.
Strips the userinfo component of ``https?://`` URLs
(``https://user:pass@host/path`` → ``https://host/path``);
``ssh://``, ``git@`` (scp-style), and local paths are left
untouched. Idempotent — and byte-identical for URLs that carry no
userinfo (no match → the input unchanged, including a ``@`` inside
the *path*, which is not userinfo). The stored row value is NOT
modified: a legacy row whose credential is still embedded in
``url`` keeps cloning with its original stored URL; this is the
output mask that keeps that credential out of every response and
the UI (the env-fallback rows get the same treatment — the env
*value* itself is untouched, only the response is masked).
"""
return _USERINFO_RE.sub(r"\1", url, count=1)
def clone_url_for(row: GitSource) -> str:
"""Phase 121, LOCKED A2 — the URL git actually clones, with the
row's credential injected ONLY here.
* ``token`` NULL/falsy → ``row.url`` verbatim: public repos and
local rows behave byte-identically to pre-phase-121, and a
legacy embedded-token row (``token`` NULL, credential in the
stored URL) keeps cloning with its ORIGINAL stored URL — the
credential keeps working;
* an ``https?://`` row with a token →
``https://x-access-token:<token>@<host>/<path>`` — any existing
userinfo in the stored URL is replaced by the column credential
(``x-access-token`` as the username: GitHub-agnostic, any host
that accepts ``https://user:token@`` treats the first component
opaquely — the task-02 assumption, task 02 step 4);
* a non-https row with a token (``ssh://``/``git@``/local path)
→ ``row.url`` unchanged + a WARNING log (a token cannot
authenticate ssh — the owner must use a deploy key/agent there;
the log names the repo via its sanitized URL, never the token).
``repo_name`` (and every other checkout-path derivation) keeps
operating on the bare ``row.url`` — the checkout directory name is
credential-free.
"""
token = row.token
if not token:
return row.url
if not row.url.startswith(("https://", "http://")):
logger.warning(
"git source %s has a stored token but a non-https? URL — "
"a token cannot authenticate ssh/git@ clones; the stored "
"URL is used as-is (configure a deploy key or SSH agent "
"for private ssh repos)",
sanitize_url(row.url),
)
return row.url
bare = sanitize_url(row.url)
return bare.replace("://", f"://x-access-token:{token}@", 1)
def normalize_credential(url: str, token: str | None) -> tuple[str, str | None]:
"""Phase 121, LOCKED A6 — the write-path credential normalizer.
If the (``https?://``-only) URL carries userinfo, it is stripped
for storage and the EMBEDDED CREDENTIAL becomes the effective
token — UNLESS the caller also sent an explicit ``token``
(non-None), which WINS (explicit beats embedded — a blank masked
field, i.e. an explicit "", is a deliberate "no credential").
Pasting the old-style ``https://user:ghp_…@host/repo.git`` URL
still works and lands token-column-clean; the caller stores the
bare URL + ``effective_token or None`` (an empty explicit token
stores NULL) and runs its duplicate check on the BARE URL, so the
same repo with a different token is still the same source
(409, not a second row).
The embedded credential is the *password* part of a
``user:pass`` userinfo (after the first colon — the password may
contain further colons), or the whole userinfo run for the
username-as-token form (``https://<token>@host/…``, the documented
GitHub shape, no colon). Clean URLs and ``ssh://``/``git@``/local
paths return ``(url, token)`` untouched — byte-identical
pre-phase behavior.
"""
match = _USERINFO_RE.match(url)
if match is None:
return url, token
userinfo = match.group(2)
user, sep, password = userinfo.partition(":")
embedded = password if sep else userinfo
effective = token if token is not None else embedded
return sanitize_url(url), effective
def effective_sources(db: Session) -> tuple[list[GitSource], Literal["db", "env"]]:
"""``(rows, origin)`` — the effective source rows (both kinds) and
app/schemas.py
+46 -2
View File
@@ -583,6 +583,14 @@ class GitSourceIn(BaseModel):
``include_hidden`` (phase 105) is optional at create time (absent →
stored ``False`` — A4).
``token`` (phase 121, LOCKED A2) is the masked private-repo
credential from the Sources page: optional at create time (absent/
None = no credential — public repo), trimmed *before* the length
constraints run (the ``_trim_url`` precedent), max 500. It is a
WRITE-ONLY field — stored in the dedicated ``git_sources.token``
column and NEVER echoed back by any output shape (``GitSourceOut``
/ ``GitSourceRow`` carry no token field by contract).
"""
kind: Literal["git", "local"] = "git"
@@ -590,6 +598,7 @@ class GitSourceIn(BaseModel):
path: str | None = Field(default=None, min_length=1, max_length=2000)
ignore_paths: list[str] | None = Field(default=None)
include_hidden: bool | None = Field(default=None)
token: str | None = Field(default=None, max_length=500)
@field_validator("url", mode="before")
@classmethod
@@ -601,6 +610,11 @@ class GitSourceIn(BaseModel):
def _trim_path(cls, v: object) -> object:
return v.strip() if isinstance(v, str) else v
@field_validator("token", mode="before")
@classmethod
def _trim_token(cls, v: object) -> object:
return v.strip() if isinstance(v, str) else v
class GitSourceOut(BaseModel):
"""One created git source as returned by ``POST`` (phase 35, task 02).
@@ -614,8 +628,18 @@ class GitSourceOut(BaseModel):
list — non-null (a row created without it reports ``[]``).
``include_hidden`` (phase 105) is the stored flag — a row created
without it reports ``False`` (A4).
There is deliberately NO ``token`` field (phase 121, LOCKED A2):
the private-repo credential is stored in the dedicated
``git_sources.token`` column and is NEVER a response field — it
never reaches the UI or any API output. ``extra="forbid"`` makes
the omission a structural contract, not an accident: constructing
this model with a ``token`` key raises, so a regression that tries
to echo the credential back cannot even build the shape.
"""
model_config = ConfigDict(extra="forbid")
id: uuid.UUID | None
url: str
added_at: datetime | None
@@ -637,8 +661,16 @@ class GitSourceRow(BaseModel):
store a list on) report ``[]``. ``include_hidden`` (phase 105) is
the row's stored flag — env-fallback rows (no DB row to store a flag
on) report ``False`` (the ``ignore_paths: []`` precedent).
There is deliberately NO ``token`` field (phase 121, LOCKED A2):
same contract as :class:`GitSourceOut` — the credential never
reaches the UI or any API output, and ``extra="forbid"`` makes the
omission structural (constructing a row with a ``token`` key
raises).
"""
model_config = ConfigDict(extra="forbid")
id: uuid.UUID | None
kind: Literal["git", "local"]
url: str
@@ -658,12 +690,24 @@ class GitSourcePatchIn(BaseModel):
normalized + A4-validated, becomes the row's whole list — empty
list clears all; every pre-phase-105 client always sends the
list, so their behavior is byte-identical). ``include_hidden``
(phase 105) when present sets the stored flag. Both absent →
200 no-op (the row is untouched).
(phase 105) when present sets the stored flag. ``token`` (phase
121, LOCKED A2) is TRI-STATE — the three-way semantics the masked
edit field depends on: **absent/None = no change** (keep the row's
stored credential), **non-empty = replace**, **empty string =
clear** (the UI offers replace; clear exists for API completeness).
Trimmed *before* the length constraints run (the ``GitSourceIn``
``_trim_token`` precedent — whitespace-only counts as a clear),
max 500. All absent → 200 no-op (the row is untouched).
"""
ignore_paths: list[str] | None = Field(default=None)
include_hidden: bool | None = Field(default=None)
token: str | None = Field(default=None, max_length=500)
@field_validator("token", mode="before")
@classmethod
def _trim_token(cls, v: object) -> object:
return v.strip() if isinstance(v, str) else v
class GitSourceList(BaseModel):
frontend/assets/git-sources.js
+46 -4
View File
@@ -203,6 +203,25 @@
* calls hideHiddenError — the phase-89 "happy path heals the error
* state" precedent).
*
* Phase 121 (task 03) — the masked token field (LOCKED A2): the add
* form gains the optional `#git-source-token` (type=password,
* autocomplete=off — a PAT is not a site credential) with the visible
* "optional — private repos" hint; the submit body is
* `(url, token) => ({ url, ...(token ? { token } : {}) })` — a blank
* token OMITS the key (None = no credential), and 201 clears BOTH
* inputs (the credential is stored — write-only: the API shapes
* carry no token field, so nothing round-trips). The per-row editor
* (the ignore-paths dialog) mirrors it: `#ignore-editor-token` with
* the placeholder "leave blank to keep the current token" — it
* always opens BLANK (there is no token field to prefill from) and
* the PATCH body includes `token` ONLY when non-blank (the tri-state:
* absent = no change, the row's stored credential is kept). Every
* display site keeps rendering `s.url` UNCHANGED — the server now
* returns bare URLs (sanitize_url), so the list cell, its title
* attribute, the remove modal, and the editor's source line are
* token-free with no per-site change; the UI must never re-embed a
* credential.
*
* Scope boundary (phase locked decisions): adding a git repo does
* NOT clone — the sync service (server-side) does that. Removing a
* source, however, performs the FULL cleanup server-side (phase 69):
@@ -239,6 +258,9 @@ export async function mount(root) {
const contentEl = root.querySelector("#git-sources-content");
const formEl = root.querySelector("#git-source-form");
const urlInput = root.querySelector("#git-source-url");
/* Phase 121: the add form's optional masked token field — blank =
no credential (the key is omitted from the POST body). */
const tokenInput = root.querySelector("#git-source-token");
const addBtn = root.querySelector("#git-source-add");
const addError = root.querySelector("#git-source-error");
/* Phase 49: the archive upload form (replaces the phase-38 local
@@ -275,6 +297,9 @@ export async function mount(root) {
const ignoreBackdrop = root.querySelector(".ignore-editor-backdrop");
const ignoreSourceEl = root.querySelector("#ignore-editor-source");
const ignoreTextarea = root.querySelector("#ignore-editor-textarea");
/* Phase 121: the editor's masked token field — blank = keep the
current token (the PATCH omits the key, the tri-state no-change). */
const ignoreTokenInput = root.querySelector("#ignore-editor-token");
const ignoreErrorEl = root.querySelector("#ignore-editor-error");
const ignoreCancelBtn = root.querySelector("#ignore-editor-cancel");
const ignoreSaveBtn = root.querySelector("#ignore-editor-save");
@@ -392,6 +417,7 @@ export async function mount(root) {
if (s.id) tr.dataset.id = s.id;
const isLocal = s.kind === "local";
// Phase 121: URLs arrive sanitized server-side — the UI must never re-embed a credential.
const value = isLocal ? (s.path ?? s.url) : s.url;
const kindLabel = isLocal ? "local" : "git";
@@ -696,6 +722,10 @@ export async function mount(root) {
// The same `value` expression makeRow uses — textContent ONLY.
if (ignoreSourceEl) ignoreSourceEl.textContent = isLocal ? s.path ?? s.url : s.url;
ignoreTextarea.value = (s.ignore_paths || []).join("\n");
// Phase 121: the token field always opens BLANK — the API has no
// token field to prefill from (LOCKED A2); blank = the PATCH
// omits the key (no change — the stored token is kept).
if (ignoreTokenInput) ignoreTokenInput.value = "";
if (ignoreErrorEl) {
ignoreErrorEl.textContent = "";
ignoreErrorEl.hidden = true; // a new attempt starts clean
@@ -719,6 +749,7 @@ export async function mount(root) {
ignoreDialog.hidden = true;
ignoreInFlight = false;
if (ignoreTextarea) ignoreTextarea.value = "";
if (ignoreTokenInput) ignoreTokenInput.value = ""; // phase 121: re-opens blank
if (ignoreErrorEl) {
ignoreErrorEl.textContent = "";
ignoreErrorEl.hidden = true;
@@ -759,6 +790,10 @@ export async function mount(root) {
// separator, not an entry (trim + drop empty; the server still
// rejects empties defensively, A4).
const lines = ignoreTextarea.value.split("\n").map((l) => l.trim()).filter(Boolean);
// Phase 121: the masked token — BLANK = the key is omitted from
// the PATCH (the tri-state: absent = no change, the row's stored
// credential is kept); non-blank replaces it.
const token = ignoreTokenInput ? ignoreTokenInput.value.trim() : "";
const t = ignoreTarget;
const value = t.kind === "local" ? (t.path ?? t.url) : t.url;
ignoreInFlight = true;
@@ -775,7 +810,7 @@ export async function mount(root) {
const r = await fetch(`/api/git-sources/${encodeURIComponent(t.id)}`, {
method: "PATCH",
headers: { "Content-Type": "application/json" },
body: JSON.stringify({ ignore_paths: lines }),
body: JSON.stringify({ ignore_paths: lines, ...(token ? { token } : {}) }),
});
if (r.ok) {
// 200: the server replaced the row's list (A5).
@@ -882,7 +917,7 @@ export async function mount(root) {
* 409/422 details are fixed generic strings (credential safety — the
* URL is never echoed). */
function wireAddForm(opts) {
const { form, input, btn, error } = opts;
const { form, input, btn, error, tokenInput } = opts;
if (!form || !input || !btn) return;
form.addEventListener("submit", async (e) => {
e.preventDefault();
@@ -900,10 +935,13 @@ export async function mount(root) {
btn.disabled = true; // §7.4: one POST per click
btn.textContent = "Adding…";
try {
// Phase 121: the masked token rides the same POST — blank =
// the key is omitted (None = no credential; LOCKED A2).
const token = tokenInput ? tokenInput.value.trim() : "";
const r = await fetch("/api/git-sources", {
method: "POST",
headers: { "Content-Type": "application/json" },
body: JSON.stringify(opts.body(value)),
body: JSON.stringify(opts.body(value, token)),
});
if (r.ok) {
let createdId = null;
@@ -913,6 +951,7 @@ export async function mount(root) {
/* the 201 body is advisory — the reload is the truth */
}
input.value = ""; // 201: the source is stored
if (tokenInput) tokenInput.value = ""; // the credential is stored (write-only)
announce(opts.addedMessage);
await loadSources(); // the new row lands in the table
focusNewRow(createdId); // a11y: land the caret on the new row
@@ -940,9 +979,12 @@ export async function mount(root) {
wireAddForm({
form: formEl,
input: urlInput,
tokenInput,
btn: addBtn,
error: addError,
body: (url) => ({ url }),
// Phase 121: the masked token is included ONLY when non-blank
// (blank = key omitted = no credential — the API's tri-state).
body: (url, token) => ({ url, ...(token ? { token } : {}) }),
emptyMessage: "Enter a git URL to add.",
failMessage: "Could not add the git source — try again.",
networkMessage: "Could not add the git source — is the app reachable?",
frontend/assets/styles.css
+35 -3
View File
@@ -2386,7 +2386,16 @@ details.thinking .thinking-text ul { margin: 0 0 0.5rem; }
#archive-upload-form:focus-within { border-color: var(--brand); box-shadow: 0 0 0 3px var(--brand-soft), var(--shadow); }
#git-source-form > label,
#archive-upload-form > label { color: var(--ink); font-weight: 600; white-space: nowrap; }
#git-source-url {
/* Phase 121 (task 03): the form-label "optional" hint — the muted
ink-soft pair (5.1:1 on the label's surface, AA) at the label's
600 weight relaxed to 400 so the hint reads as secondary (it
qualifies the name, it is not the name); small, inline. */
.field-hint { color: var(--ink-soft); font-weight: 400; font-size: 0.8rem; }
/* #git-source-token (phase 121): the optional masked private-repo
credential — the URL input's treatment VERBATIM (mono, >=44px
target, brand focus); type=password masks the value in display. */
#git-source-url,
#git-source-token {
flex: 1;
min-width: 14rem;
min-height: 44px;
@@ -2398,8 +2407,10 @@ details.thinking .thinking-text ul { margin: 0 0 0.5rem; }
border-radius: var(--radius-sm);
padding: 0.45rem 0.7rem;
}
#git-source-url::placeholder { color: var(--ink-soft); }
#git-source-url:focus-visible { outline-offset: 0; border-color: var(--brand); }
#git-source-url::placeholder,
#git-source-token::placeholder { color: var(--ink-soft); }
#git-source-url:focus-visible,
#git-source-token:focus-visible { outline-offset: 0; border-color: var(--brand); }
#git-source-add,
#archive-upload-btn {
display: inline-flex;
@@ -2802,6 +2813,26 @@ details.thinking .thinking-text ul { margin: 0 0 0.5rem; }
resize: vertical;
}
/* Phase 121 (task 03): the editor's masked token field — the
textarea's box language (full panel width, --bg fill, the line
border, mono) at the 44px touch floor; the label's top margin keeps
the field pair off the textarea. type=password + autocomplete=off
live in the markup (a PAT is not a site credential — no browser
save offer); the focus ring is the global 3px outline rule. */
.ignore-editor-token-label { margin-top: 0.9rem; }
.ignore-editor-token {
display: block;
width: 100%;
min-height: 44px;
padding: 0.55rem 0.65rem;
font-family: var(--mono);
font-size: 0.85rem;
color: var(--ink);
background: var(--bg);
border: 1px solid var(--line);
border-radius: var(--radius-sm);
}
/* The in-dialog failure line (role=alert): the err pair (err-ink on
err-bg 9.3:1, the err-line border) — the .remove-confirm-error
language. */
@@ -4943,6 +4974,7 @@ details.thinking .thinking-text ul { margin: 0 0 0.5rem; }
#git-source-form > label,
#archive-upload-form > label { white-space: normal; }
#git-source-url,
#git-source-token,
#archive-upload-file { min-width: 0; }
#git-source-add,
#archive-upload-btn { width: 100%; }
frontend/index.html
+37 -5
View File
@@ -625,10 +625,11 @@
to the database.
</p>
<!-- Add form: visible label + mono URL input + brand button
(dark ink on brand 5.2:1). §7.4 never-stale: the button
disables + relabels "Adding…" while the POST is in flight
and re-enables on success AND failure (the input is kept
<!-- Add form: visible label + mono URL input + the optional
masked token field (phase 121) + brand button (dark ink
on brand 5.2:1). §7.4 never-stale: the button disables +
relabels "Adding…" while the POST is in flight and
re-enables on success AND failure (the inputs are kept
on failure, same as the tuning forms). -->
<form id="git-source-form">
<label for="git-source-url">Add a git source</label>
@@ -641,6 +642,22 @@
placeholder="https://github.com/you/your-repo.git"
required
>
<!-- Phase 121 (task 03, LOCKED A2): the optional masked
private-repo credential — type=password +
autocomplete=off (a PAT is not a site credential: no
browser save offer). Blank = the POST omits the key
(None = no credential); the token is write-only — it
never round-trips (the API shapes carry no token
field). -->
<label for="git-source-token">Token <span class="field-hint">optional — private repos</span></label>
<input
id="git-source-token"
name="token"
type="password"
maxlength="500"
autocomplete="off"
placeholder="ghp_… or another PAT"
>
<button type="submit" id="git-source-add">Add source</button>
<p class="git-source-error" id="git-source-error" role="alert" hidden></p>
</form>
@@ -784,12 +801,16 @@
while the PATCH is out). Stored rows only (A3) — the
per-row button lives in git-sources.js's makeRow; this
is the page-local dialog it opens. -->
<!-- Phase 121 (task 03): the per-row editor gains the
optional masked token field (LOCKED A2) — the title
widens from "Ignored files and folders" to the dialog's
actual scope (the row's settings). -->
<div class="ignore-editor" id="ignore-editor-dialog" role="alertdialog"
aria-modal="true" aria-labelledby="ignore-editor-title"
aria-describedby="ignore-editor-copy" hidden>
<div class="ignore-editor-backdrop" aria-hidden="true"></div>
<div class="ignore-editor-panel">
<h2 class="ignore-editor-title" id="ignore-editor-title">Ignored files and folders</h2>
<h2 class="ignore-editor-title" id="ignore-editor-title">Source settings</h2>
<code class="ignore-editor-source" id="ignore-editor-source"></code>
<p class="ignore-editor-copy" id="ignore-editor-copy">
One path per line. A file is ignored when its path in the
@@ -801,6 +822,17 @@
<textarea class="ignore-editor-textarea" id="ignore-editor-textarea"
rows="6" spellcheck="false"
placeholder="my/files/"></textarea>
<!-- Phase 121 (task 03, LOCKED A2): the masked
private-repo credential — type=password +
autocomplete=off (a PAT is not a site credential).
BLANK = the PATCH omits the key (the tri-state: no
change — the row's stored token is kept). The field
is NEVER prefilled (the API has no token field to
read it from). -->
<label class="ignore-editor-label ignore-editor-token-label" for="ignore-editor-token">Token <span class="field-hint">optional — private repos</span></label>
<input class="ignore-editor-token" id="ignore-editor-token"
type="password" maxlength="500" autocomplete="off"
placeholder="leave blank to keep the current token">
<p class="ignore-editor-error" id="ignore-editor-error" role="alert" hidden></p>
<div class="ignore-editor-actions">
<button type="button" class="ignore-editor-btn ignore-editor-cancel"
scripts/import_docs.py
+15 -6
View File
@@ -19,10 +19,15 @@ precedence order:
(first run, full history — no ``--depth``; an existing shallow
checkout is unshallowed first, phase 107) or fast-forwarded
(``git pull --ff-only``) into ``BOR_SOURCES_DIR/<repo-name>/``
(default ``~/bor-sources``); local rows are the existing directories
themselves, walked directly. A failing clone/pull — or a local
directory that is missing at run time — aborts the whole run
*before* anything is imported.
(default ``~/bor-sources``) — with the phase-121 clone URL
(:func:`app.rag.git_sources.clone_url_for`): the row's ``token``
column injected as ``https://x-access-token:<token>@…`` only for
https? rows, NULL token → the bare stored URL verbatim (public
repos and legacy embedded-token rows clone exactly as before);
the checkout name stays on the bare URL (credential-free); local
rows are the existing directories themselves, walked directly. A
failing clone/pull — or a local directory that is missing at run
time — aborts the whole run *before* anything is imported.
3. Fallback — the legacy ``DEFAULT_SOURCES`` (``~/Homelab`` +
``~/Deployments``), kept for backwards compatibility (reached only
while both the table and ``BOR_GIT_SOURCES`` are empty).
@@ -113,7 +118,7 @@ from app.core.logging import configure_logging
from app.db import SessionLocal
from app.models import KbOverview
from app.rag.folder_summaries import generate_folder_summaries, missing_folder_summaries
from app.rag.git_sources import effective_sources
from app.rag.git_sources import clone_url_for, effective_sources
from app.rag.importer import ImportSummary, import_sources
from app.rag.llm import LLMClient
from app.rag.overview import regenerate_overview
@@ -236,7 +241,11 @@ def _resolve_sources(
doc_dates_by_root: dict[str, dict[str, datetime]] = {}
for row in rows:
if row.kind == "git":
root = clone_or_pull(row.url, sources_root / repo_name(row.url))
# Phase 121: the token column is injected into the clone
# URL ONLY here (clone_url_for — NULL token → the bare
# stored URL verbatim); repo_name stays on the bare URL
# so the checkout directory name is credential-free.
root = clone_or_pull(clone_url_for(row), sources_root / repo_name(row.url))
# Phase 106 (D2): the checkout's per-file last-commit
# dates, keyed by the SAME root string the importer
# sees; local rows contribute nothing (mtime fallback).
tests/e2e/test_git_source_tokens.py
+429
View File
@@ -0,0 +1,429 @@
"""Phase 121 story E2E (Playwright): private git sources — a token that
never reaches the UI or the API (LOCKED A2/A6).
Source: ``TODO.md:5`` — "Need a way to add private repos without exposing
the token in the UI (like when adding an https repo
``https://myuser:ghp_xxx@github.com/myuser/my-private-repo.git``)".
Run in isolation (DB must be up: ``podman compose up -d db``):
uv run pytest tests/e2e/test_git_source_tokens.py -v --no-cov
**No git, no network** — nothing in this suite triggers a sync (the
clone-time credential injection is pinned at the integration level with a
mocked ``clone_or_pull``); this suite drives the real Sources page and
the real admin-only CRUD API and asserts the credential contract on every
surface the owner can see: the rendered row, the page text, every
``title`` attribute, the page HTML, and the raw JSON of
``GET /api/git-sources`` (via the logged-in page context's request
client — the same cookie as the UI).
Per-module app env (the conftest pattern, module-scoped — as in
``test_git_sources_admin.py``): this story's app boots on its own port
(a same-port second uvicorn would die on bind and drive the wrong
server) with ``BOR_GIT_SOURCES`` pinned EMPTY (the phase-61 leak guard —
the operator's local gitignored ``.env`` carries the owner's real source
list, and process env ranks above the ``.env`` file; a leaked env list
would flip the empty-after-removal assertions) and ``BOR_SOURCES_DIR`` /
``BOR_UPLOAD_DIR`` pinned to fresh tmp dirs (the cleanup DELETE must
never aim a rmtree at the operator's real checkouts — the integration
``_settings`` pattern). The mock-calibrated lexical floor is pinned
like the conftest (the 0.30 mock threshold + the code-default 0.35 floor
violate the startup validator).
Contract under test (the phase's completion criteria, UI side):
* add a private repo through the Sources UI (bare URL + the masked
``#git-source-token`` field) → 201 → the row renders the BARE URL
(mono cell + the cell's ``title`` attribute), the token is absent from
``document.body.innerText``, every ``title`` attribute, the page HTML,
and the ``GET /api/git-sources`` raw JSON; the DB row stores the bare
URL + the token column (the DB is the trusted store); both inputs
clear and the button re-enables (§7.4 never-stale);
* pasting the old-style embedded-token URL (``user:token@``, no token
field — the TODO L5 shape) is accepted and normalized server-side
(LOCKED A6): stored bare + the token column populated, and the token
(and the username) is nowhere in the UI or the API;
* the per-row editor's ``#ignore-editor-token`` is ``type=password`` and
opens BLANK (a password is never echoed back — there is no token
field to prefill from); saving it blank sends a PATCH that OMITS the
key (the tri-state no-change), gets 200, and the row's stored token is
KEPT (DB assertion) — the row is intact;
* removing the source (the phase-69 confirmation modal) → 204, the list
is empty again, and the token is gone from every surface.
Test → mapping:
1. ``test_admin_add_private_repo_token_stays_out_of_ui_and_api``
2. ``test_admin_embedded_token_url_paste_is_normalized_and_hidden``
3. ``test_admin_editor_token_opens_blank_and_save_keeps_token``
4. ``test_admin_remove_private_repo_leaves_empty_list``
"""
from __future__ import annotations
import json
import os
import subprocess
import sys
import tempfile
from collections.abc import Iterator
from pathlib import Path
from typing import Any
import pytest
from playwright.sync_api import Page, expect
from sqlalchemy import select, text
from app.db import SessionLocal
from app.models import GitSource
from e2e.auth_helpers import login
from e2e.conftest import ADMIN_PASSWORD, SESSION_SECRET, USE_REAL_LLM, _wait_http
REPO = Path(__file__).resolve().parents[2]
# Phase 79 (task 04, full inventory): the conftest session app owns its
# port in a combined run — this module app binds its own port instead
# (env-overridable, as in test_git_sources_admin.py).
APP_PORT = int(os.environ.get("E2E_APP_PORT_GITTOKEN", "8146"))
APP_URL = f"http://127.0.0.1:{APP_PORT}"
GIT_SOURCES_URL = "/git-sources.html"
#: Deterministic fixtures: a private repo, a distinctive fake token (the
#: "nowhere" assertions key on this exact string), and the old-style
#: embedded-token paste of a SECOND repo (the TODO L5 shape).
PRIVATE_URL = "https://github.com/acme/e2e-private-token.git"
TOKEN = "ghp_e2esecrettoken0123456789"
LEGACY_URL = f"https://myuser:{TOKEN}@github.com/acme/e2e-legacy.git"
LEGACY_BARE_URL = "https://github.com/acme/e2e-legacy.git"
# ---------------------------------------------------------------------------
# Fixtures
# ---------------------------------------------------------------------------
@pytest.fixture(scope="module")
def app_server(mock_llm: int) -> Iterator[str]:
"""The real app under test — per-module env: ``BOR_GIT_SOURCES``
pinned EMPTY (the phase-61 leak guard — see the module docstring)
and ``BOR_SOURCES_DIR`` / ``BOR_UPLOAD_DIR`` pinned to fresh tmp
dirs (the cleanup DELETE never rmtrees the operator's real
checkouts). No git, no sync in this suite."""
env = dict(os.environ)
env.pop("DEBUGPY", None)
env["BOR_ENVIRONMENT"] = "e2e"
env["BOR_STATIC_DIR"] = str(REPO / "frontend")
env["BOR_LLM_BASE_URL"] = (
"https://aipi.reeseapps.com/v1"
if USE_REAL_LLM
else f"http://127.0.0.1:{mock_llm}/v1"
)
# Mock-calibrated gate (conftest pattern) — no chat turn is sent,
# but the app boots with the same env shape. The floor is pinned
# explicitly: the code default (0.35) violates the startup
# validator against the mock threshold (0.30) and would refuse the
# boot if the operator's .env did not carry a valid one.
env["BOR_RELEVANCE_THRESHOLD"] = "0.30"
env["BOR_LEXICAL_SUPPORT_FLOOR"] = "0.15"
env.setdefault(
"BOR_DATABASE_URL",
"postgresql+psycopg://reese:reese@localhost:5432/brain_of_reese",
)
# Phase 16: admin auth must be set or create_app() refuses to boot.
env["BOR_ADMIN_PASSWORD"] = ADMIN_PASSWORD
env["BOR_SESSION_SECRET"] = SESSION_SECRET
# The empty-table env fallback list — pinned EMPTY: the operator's
# local (gitignored) .env must not leak the owner's real sources
# into the empty-state assertions.
env["BOR_GIT_SOURCES"] = ""
# Fresh scratch dirs: the cleanup DELETE (test 4) must never aim a
# rmtree at the operator's real checkouts (the integration
# ``_settings`` pattern).
scratch = Path(tempfile.mkdtemp(prefix="bor-e2e-gittoken-"))
env["BOR_SOURCES_DIR"] = str(scratch / "sources")
env["BOR_UPLOAD_DIR"] = str(scratch / "uploads")
proc = subprocess.Popen(
[sys.executable, "-m", "uvicorn", "app.main:app",
"--host", "127.0.0.1", "--port", str(APP_PORT), "--log-level", "warning"],
cwd=REPO,
env=env,
)
try:
_wait_http(f"{APP_URL}/api/health")
yield APP_URL
finally:
proc.terminate()
try:
proc.wait(timeout=10)
except subprocess.TimeoutExpired:
proc.kill()
@pytest.fixture(scope="module")
def app_url(app_server: str) -> str:
return app_server
def _truncate_git_sources() -> None:
with SessionLocal() as db:
db.execute(text("TRUNCATE git_sources"))
db.commit()
@pytest.fixture(autouse=True)
def _clean_git_sources(db_ready: None) -> Iterator[None]:
"""Fresh ``git_sources`` table per test — this suite owns the table
(the E2E isolation pattern, as in ``test_git_sources_admin.py``);
emptied BOTH before and after every test (the suites share one
Postgres and run in isolation — a leftover row would flip another
suite's app from the env fallback to the DB list)."""
_truncate_git_sources()
yield
_truncate_git_sources()
# ---------------------------------------------------------------------------
# Helpers
# ---------------------------------------------------------------------------
def _seed_token_row(url: str, token: str) -> None:
"""Store a token row directly (deterministic; the page loads the
list on boot, so seed BEFORE navigating)."""
with SessionLocal() as db:
db.add(GitSource(url=url, token=token))
db.commit()
def _stored_row() -> tuple[str, str | None]:
"""The stored row's ``(url, token)`` — read INSIDE the session
(detached instances would defer-load on attribute access)."""
with SessionLocal() as db:
row = db.scalars(select(GitSource)).one()
return row.url, row.token
def _admin_git_sources_page(page: Page, app_url: str) -> None:
"""Real form login landing on the git sources page (admin settled:
Sign out visible, the manager revealed by the page module)."""
login(page, app_url, next=GIT_SOURCES_URL)
expect(page).to_have_url(app_url + GIT_SOURCES_URL, timeout=30_000)
expect(page.locator("#sign-out-btn")).to_be_visible(timeout=15_000)
expect(page.locator("#git-sources-gate")).to_be_hidden()
expect(page.locator("#git-sources-content")).to_be_visible()
def _assert_token_nowhere(page: Page, app_url: str, token: str) -> None:
"""The phase's credential contract (LOCKED A2): the token is absent
from EVERY surface the owner can see — the rendered body text,
every ``title`` attribute, the whole page HTML (every serialized
attribute value), and the raw JSON text of ``GET /api/git-sources``
(the page context's request client carries the logged-in cookie)."""
body_text = page.evaluate("() => document.body.innerText")
assert token not in body_text, "the token leaks into the page text"
titles = page.evaluate(
"() => [...document.querySelectorAll('[title]')].map((el) => el.getAttribute('title'))"
)
for t in titles:
assert t is None or token not in t, f"the token leaks into a title attribute: {t!r}"
assert token not in page.content(), "the token leaks into the page HTML"
r = page.request.get(f"{app_url}/api/git-sources")
assert r.status == 200, r.text
assert token not in r.text(), "the token leaks into GET /api/git-sources"
# ---------------------------------------------------------------------------
# 1. Add: bare URL + masked token → the row is bare, the token is
# nowhere (page text, title attributes, HTML, API JSON)
# ---------------------------------------------------------------------------
def test_admin_add_private_repo_token_stays_out_of_ui_and_api(
page: Page, app_url: str, db_ready: None
) -> None:
page.set_default_timeout(30_000)
_admin_git_sources_page(page, app_url)
expect(page.locator("#git-sources-tbody tr")).to_have_count(0)
page.fill("#git-source-url", PRIVATE_URL)
page.fill("#git-source-token", TOKEN)
page.click("#git-source-add")
# 201 → the row lands rendering the BARE URL — the mono code cell
# AND the cell's hover title attribute are both bare.
row = page.locator("#git-sources-tbody tr", has_text=PRIVATE_URL)
expect(row).to_have_count(1, timeout=30_000)
expect(row.locator("td.git-source-url-cell code")).to_have_text(PRIVATE_URL)
expect(row.locator("td.git-source-url-cell")).to_have_attribute("title", PRIVATE_URL)
# Never-stale (§7.4): BOTH inputs clear (the credential is stored —
# write-only, never round-trips), the button re-enables with its
# idle label, no inline error.
expect(page.locator("#git-source-url")).to_have_value("")
expect(page.locator("#git-source-token")).to_have_value("")
expect(page.locator("#git-source-add")).to_be_enabled()
expect(page.locator("#git-source-add")).to_have_text("Add source")
expect(page.locator("#git-source-error")).to_be_hidden()
# The DB is the trusted store (LOCKED A2): bare URL + the token
# column — the token lives in exactly one place.
stored_url, stored_token = _stored_row()
assert stored_url == PRIVATE_URL
assert stored_token == TOKEN
_assert_token_nowhere(page, app_url, TOKEN)
# ---------------------------------------------------------------------------
# 2. Paste the old-style embedded-token URL (TODO L5): normalized
# server-side (LOCKED A6), token + username nowhere
# ---------------------------------------------------------------------------
def test_admin_embedded_token_url_paste_is_normalized_and_hidden(
page: Page, app_url: str, db_ready: None
) -> None:
page.set_default_timeout(30_000)
_admin_git_sources_page(page, app_url)
expect(page.locator("#git-sources-tbody tr")).to_have_count(0)
# The exact TODO L5 paste: ``user:token@`` in the URL, the masked
# token field BLANK (the key is omitted from the POST).
page.fill("#git-source-url", LEGACY_URL)
page.click("#git-source-add")
# 201 → the row is the BARE URL (the userinfo was stripped on
# write), in the code cell AND the title attribute.
row = page.locator("#git-sources-tbody tr", has_text=LEGACY_BARE_URL)
expect(row).to_have_count(1, timeout=30_000)
expect(row.locator("td.git-source-url-cell code")).to_have_text(LEGACY_BARE_URL)
expect(row.locator("td.git-source-url-cell")).to_have_attribute("title", LEGACY_BARE_URL)
expect(page.locator("#git-source-error")).to_be_hidden()
# The paste is normalized, not rejected: stored bare + the embedded
# PASSWORD part in the token column (the username is an identifier,
# not the credential).
stored_url, stored_token = _stored_row()
assert stored_url == LEGACY_BARE_URL
assert stored_token == TOKEN
# The token is nowhere — and the username is gone from the page
# text too (the whole userinfo was stripped, not just the secret).
_assert_token_nowhere(page, app_url, TOKEN)
body_text = page.evaluate("() => document.body.innerText")
assert "myuser" not in body_text, "the username leaks into the page text"
# ---------------------------------------------------------------------------
# 3. Edit: the editor's token field opens BLANK (never prefilled) and a
# blank save is a 200 no-change — the stored token is kept
# ---------------------------------------------------------------------------
def test_admin_editor_token_opens_blank_and_save_keeps_token(
page: Page, app_url: str, db_ready: None
) -> None:
page.set_default_timeout(30_000)
_seed_token_row(PRIVATE_URL, TOKEN) # before the page loads its list
_admin_git_sources_page(page, app_url)
row = page.locator("#git-sources-tbody tr", has_text=PRIVATE_URL)
expect(row).to_have_count(1, timeout=30_000)
# Open the per-row editor ("Ignore paths" — the row's settings).
row.locator(".git-source-ignore").click()
dialog = page.locator("#ignore-editor-dialog")
expect(dialog).to_be_visible(timeout=30_000)
expect(page.locator("#ignore-editor-source")).to_have_text(PRIVATE_URL)
# The token field: masked, and BLANK — a password must never be
# echoed back (there is no token field to prefill from — LOCKED A2).
tok = page.locator("#ignore-editor-token")
expect(tok).to_have_value("")
assert tok.get_attribute("type") == "password"
patch_bodies: list[dict[str, Any] | None] = []
patch_statuses: list[int] = []
def track_request(r: Any) -> None:
if r.method == "PATCH" and "/api/git-sources/" in r.url:
try:
patch_bodies.append(json.loads(r.post_data))
except (TypeError, ValueError):
patch_bodies.append(None)
def track_response(r: Any) -> None:
if r.request.method == "PATCH" and "/api/git-sources/" in r.url:
patch_statuses.append(r.status)
page.on("request", track_request)
page.on("response", track_response)
try:
# Save with the token field blank → 200 (no 4xx), the dialog
# closes, no inline error, the row is intact.
page.click("#ignore-editor-save")
expect(dialog).to_be_hidden(timeout=30_000)
expect(page.locator("#ignore-editor-error")).to_be_hidden()
expect(page.locator("#git-sources-tbody tr", has_text=PRIVATE_URL)).to_have_count(1)
finally:
page.remove_listener("request", track_request)
page.remove_listener("response", track_response)
# One PATCH, 200, and the body OMITS the token key (blank = the
# tri-state no-change — the key is not sent, let alone empty).
assert patch_statuses == [200], f"the blank-token save did not 200: {patch_statuses}"
assert len(patch_bodies) == 1 and patch_bodies[0] is not None
assert "token" not in patch_bodies[0], "a blank token must OMIT the PATCH key"
assert patch_bodies[0] == {"ignore_paths": []}
# The row is intact and the stored token is KEPT (the no-change
# tri-state, verified in the trusted store).
stored_url, stored_token = _stored_row()
assert stored_url == PRIVATE_URL
assert stored_token == TOKEN
_assert_token_nowhere(page, app_url, TOKEN)
# ---------------------------------------------------------------------------
# 4. Cleanup: the confirmation-modal removal → 204, the list is empty
# again, and the token is gone from every surface
# ---------------------------------------------------------------------------
def test_admin_remove_private_repo_leaves_empty_list(
page: Page, app_url: str, db_ready: None
) -> None:
page.set_default_timeout(30_000)
_seed_token_row(PRIVATE_URL, TOKEN) # before the page loads its list
_admin_git_sources_page(page, app_url)
row = page.locator("#git-sources-tbody tr", has_text=PRIVATE_URL)
expect(row).to_have_count(1, timeout=30_000)
delete_statuses: list[int] = []
def track_response(r: Any) -> None:
if r.request.method == "DELETE" and "/api/git-sources/" in r.url:
delete_statuses.append(r.status)
page.on("response", track_response)
try:
row.locator(".git-source-remove").click()
dialog = page.locator("#remove-confirm-dialog")
expect(dialog).to_be_visible(timeout=30_000)
# The modal names the source BARE (the same sanitized value the
# row shows — a credential is never rendered here either).
expect(page.locator("#remove-confirm-source")).to_have_text(PRIVATE_URL)
page.click("#remove-confirm-remove")
expect(page.locator("#git-sources-tbody tr")).to_have_count(0, timeout=30_000)
expect(dialog).to_be_hidden()
finally:
page.remove_listener("response", track_response)
# Total removal: one DELETE, 204, and the list is empty again —
# the API agrees (and the row's token left with the row).
assert delete_statuses == [204], f"the removal did not 204: {delete_statuses}"
r = page.request.get(f"{app_url}/api/git-sources")
assert r.status == 200, r.text
assert r.json()["sources"] == []
assert TOKEN not in r.text()
_assert_token_nowhere(page, app_url, TOKEN)
tests/integration/test_git_sources_api.py
+472
View File
@@ -45,9 +45,23 @@ Contract under test:
unchanged.
``git_sources`` is global state: truncated around every test.
Phase 121 (task 02 — clone-time credential + output sanitization):
POST normalizes an old-style embedded ``user:pass@`` URL into the bare
URL + ``token`` column (explicit ``token`` wins — LOCKED A6), the
duplicate check runs on the bare URL, the PATCH token is tri-state
(absent/None = no change, non-empty = replace, "" = clear) and
re-normalizes a legacy row's URL, and every response (list DB rows,
list env rows, POST 201, PATCH 200) is token-free — asserted on the
RAW response text — while a token row's SYNC clone receives the
injected ``https://x-access-token:<token>@…`` URL with a
credential-free checkout path (mock ``clone_or_pull``) and a legacy
row's clone still receives its ORIGINAL stored URL (the credential
keeps working).
"""
from __future__ import annotations
import time
import uuid
from collections.abc import Iterator
from datetime import UTC, datetime, timedelta
@@ -60,8 +74,12 @@ from sqlalchemy import select, text
from sqlalchemy.orm import Session
from app.api import git_sources as git_sources_api
from app.api import sync as sync_api
from app.config import Settings
from app.main import app as fastapi_app
from app.models import GitSource
from app.rag import git_sources as git_sources_resolver
from app.rag.importer import ImportSummary
@pytest.fixture(autouse=True)
@@ -949,3 +967,457 @@ def test_patch_422_does_not_half_apply_flag(admin_client: TestClient) -> None:
row = admin_client.get("/api/git-sources").json()["sources"][0]
assert row["ignore_paths"] == ["keep"]
assert row["include_hidden"] is False
# --- token: write-path normalization (phase 121, task 02) ------------------
TOKEN = "ghp_phase121secrettoken"
def test_post_with_token_stores_column_and_never_echoes(
admin_client: TestClient, db: Session
) -> None:
"""Bare URL + masked token → the token is stored in the dedicated
column and appears in NO API response — asserted on the RAW text of
both the 201 and the GET list (the response shapes have no token
field by contract; this pins that nothing else smuggles it out)."""
r = admin_client.post(
"/api/git-sources",
json={"url": "https://github.com/owner/private.git", "token": TOKEN},
)
assert r.status_code == 201, r.text
body = r.json()
assert body["url"] == "https://github.com/owner/private.git"
assert set(body) == {"id", "url", "added_at", "ignore_paths", "include_hidden"}
assert TOKEN not in r.text
row = db.scalars(select(GitSource)).one()
assert row.url == "https://github.com/owner/private.git" # bare
assert row.token == TOKEN # the column, not the URL
r = admin_client.get("/api/git-sources")
assert r.status_code == 200
assert TOKEN not in r.text
assert r.json()["sources"][0]["url"] == "https://github.com/owner/private.git"
def test_post_embedded_token_url_normalizes_to_bare_plus_column(
admin_client: TestClient, db: Session
) -> None:
"""The TODO L5 paste — old-style ``user:token@`` URL with no token
field: stored BARE, the embedded PASSWORD part moved to the token
column, and every response is token-free (raw text)."""
r = admin_client.post(
"/api/git-sources",
json={"url": f"https://myuser:{TOKEN}@github.com/owner/private-repo.git"},
)
assert r.status_code == 201, r.text
assert r.json()["url"] == "https://github.com/owner/private-repo.git"
assert TOKEN not in r.text
assert "myuser" not in r.text
row = db.scalars(select(GitSource)).one()
assert row.url == "https://github.com/owner/private-repo.git"
assert row.token == TOKEN # the password part, not the whole userinfo
r = admin_client.get("/api/git-sources")
assert TOKEN not in r.text
assert r.json()["sources"][0]["url"] == "https://github.com/owner/private-repo.git"
def test_post_explicit_token_wins_over_embedded(
admin_client: TestClient, db: Session
) -> None:
"""LOCKED A6 — the masked field and the pasted URL disagree: the
explicit field is the intent and beats the embedded credential."""
r = admin_client.post(
"/api/git-sources",
json={
"url": f"https://myuser:{TOKEN}@github.com/owner/private.git",
"token": "ghp_explicitwins",
},
)
assert r.status_code == 201, r.text
assert r.json()["url"] == "https://github.com/owner/private.git"
row = db.scalars(select(GitSource)).one()
assert row.url == "https://github.com/owner/private.git"
assert row.token == "ghp_explicitwins"
def test_post_username_as_token_form_moves_whole_run(
admin_client: TestClient, db: Session
) -> None:
"""The documented GitHub shape (no colon) — the whole userinfo run
is the credential."""
r = admin_client.post(
"/api/git-sources",
json={"url": f"https://{TOKEN}@github.com/owner/private.git"},
)
assert r.status_code == 201, r.text
row = db.scalars(select(GitSource)).one()
assert row.url == "https://github.com/owner/private.git"
assert row.token == TOKEN
def test_post_same_repo_different_token_is_409_on_bare_url(
admin_client: TestClient, db: Session
) -> None:
"""The duplicate check runs on the NORMALIZED bare URL: the same
repo pasted with a different credential is the same source — 409,
not a second row (both via embedded and via explicit token)."""
assert (
admin_client.post(
"/api/git-sources",
json={"url": f"https://user1:{TOKEN}@github.com/owner/dup.git"},
).status_code
== 201
)
r = admin_client.post(
"/api/git-sources",
json={"url": "https://user2:other-cred@github.com/owner/dup.git"},
)
assert r.status_code == 409
assert r.json()["detail"] == "a git source with this URL already exists"
assert TOKEN not in r.text
# And the bare form of the same repo 409s too.
r = admin_client.post(
"/api/git-sources",
json={"url": "https://github.com/owner/dup.git", "token": "another"},
)
assert r.status_code == 409
assert db.execute(text("SELECT count(*) FROM git_sources")).scalar_one() == 1
def test_post_local_kind_token_is_inert_and_never_echoed(
admin_client: TestClient, tmp_path: Path, db: Session
) -> None:
"""Local-kind sources have no URL credential (the design does not
touch them): a token on a local row is stored inert (never used —
local rows are walked, not cloned) and, like on git rows, never
echoed."""
real_dir = tmp_path / "local-tok"
real_dir.mkdir()
r = admin_client.post(
"/api/git-sources",
json={"kind": "local", "path": str(real_dir), "token": TOKEN},
)
assert r.status_code == 201, r.text
assert TOKEN not in r.text
assert r.json()["url"] == str(real_dir)
row = db.scalars(select(GitSource)).one()
assert row.token == TOKEN # inert — clone_url_for never sees it
assert TOKEN not in admin_client.get("/api/git-sources").text
# --- token: PATCH tri-state (phase 121, task 02) ---------------------------
def test_patch_token_replace_clear_and_noop(
admin_client: TestClient, db: Session
) -> None:
"""The tri-state on a clean-URL row: absent/None = no change,
non-empty = replace, "" = clear (stored NULL) — the token is never
in any response (raw text)."""
created = admin_client.post(
"/api/git-sources",
json={"url": "https://github.com/owner/patch.git", "token": TOKEN},
)
assert created.status_code == 201
sid = created.json()["id"]
# Absent (and explicit None) = no change.
# The endpoint commits in its own session — expire this one's
# identity map before reading the row back (the house pattern for
# cross-session reads).
def stored() -> GitSource:
db.expire_all()
row = db.get(GitSource, uuid.UUID(sid))
assert row is not None
return row
r = admin_client.patch(f"/api/git-sources/{sid}", json={"ignore_paths": ["a"]})
assert r.status_code == 200, r.text
assert TOKEN not in r.text
assert r.json()["url"] == "https://github.com/owner/patch.git"
assert stored().token == TOKEN
r = admin_client.patch(
f"/api/git-sources/{sid}", json={"ignore_paths": None, "token": None}
)
assert r.status_code == 200
assert stored().token == TOKEN
# Non-empty = replace (the URL is untouched — a clean URL comes
# back byte-identical).
r = admin_client.patch(f"/api/git-sources/{sid}", json={"token": "ghp_replaced"})
assert r.status_code == 200, r.text
assert TOKEN not in r.text
assert "ghp_replaced" not in r.text
assert r.json()["url"] == "https://github.com/owner/patch.git"
row = stored()
assert row.url == "https://github.com/owner/patch.git"
assert row.token == "ghp_replaced"
# "" = clear — stored NULL.
r = admin_client.patch(f"/api/git-sources/{sid}", json={"token": ""})
assert r.status_code == 200, r.text
row = stored()
assert row.token is None
assert r.json()["url"] == "https://github.com/owner/patch.git"
def test_patch_token_on_legacy_row_renormalizes_url(
admin_client: TestClient, db: Session
) -> None:
"""A pre-phase row (credential embedded in the stored URL, token
NULL) gets its userinfo stripped (moved to the column) the first
time an explicit credential is written — and a "" clear also moves
it (a cleared credential is gone from both the column and the URL;
the owner explicitly asked for no credential)."""
legacy = f"https://user:{TOKEN}@github.com/owner/legacy.git"
db.add(GitSource(url=legacy))
db.commit()
row = db.scalars(select(GitSource)).one()
sid = row.id
r = admin_client.patch(f"/api/git-sources/{sid}", json={"token": "ghp_new"})
assert r.status_code == 200, r.text
assert TOKEN not in r.text
assert "ghp_new" not in r.text
assert r.json()["url"] == "https://github.com/owner/legacy.git"
db.expire_all() # the endpoint committed in its own session
row = db.get(GitSource, sid)
assert row is not None
assert row.url == "https://github.com/owner/legacy.git" # normalized bare
assert row.token == "ghp_new"
# "" clears the (now column) credential — the URL stays bare.
r = admin_client.patch(f"/api/git-sources/{sid}", json={"token": ""})
assert r.status_code == 200, r.text
db.expire_all()
row = db.get(GitSource, sid)
assert row is not None
assert row.url == "https://github.com/owner/legacy.git"
assert row.token is None
def test_patch_token_409_backstop_on_renormalized_collision(
admin_client: TestClient, db: Session
) -> None:
"""A legacy embedded row + a bare row for the same repo can only
coexist pre-phase; re-normalizing the legacy row's URL on a token
PATCH makes the stored URLs collide — the unique index yields the
generic 409 (never a 500) and the failed PATCH leaves the row
untouched."""
legacy = f"https://user:{TOKEN}@github.com/owner/collide.git"
bare = "https://github.com/owner/collide.git"
db.add_all([GitSource(url=legacy), GitSource(url=bare)])
db.commit()
legacy_row = db.scalars(select(GitSource).where(GitSource.url == legacy)).one()
r = admin_client.patch(f"/api/git-sources/{legacy_row.id}", json={"token": "ghp_x"})
assert r.status_code == 409
assert r.json()["detail"] == "a git source with this URL already exists"
assert TOKEN not in r.text
# The rollback left the legacy row exactly as it was.
db.expire_all()
row = db.get(GitSource, legacy_row.id)
assert row is not None
assert row.url == legacy
assert row.token is None
# --- token: output sanitization (phase 121, task 02) ------------------------
def test_get_masks_legacy_embedded_token_row(admin_client: TestClient, db: Session) -> None:
"""Completion criterion: a legacy row (token embedded in the
stored URL, token column NULL) clones with its original URL but
its API output is token-free — the stored DB value is untouched,
the response is bare."""
legacy = f"https://user:{TOKEN}@github.com/owner/legacy.git"
db.add(GitSource(url=legacy))
db.commit()
r = admin_client.get("/api/git-sources")
assert r.status_code == 200
assert TOKEN not in r.text
assert "user:" not in r.text
assert r.json()["sources"][0]["url"] == "https://github.com/owner/legacy.git"
# The stored value is untouched (the clone still authenticates).
assert db.scalars(select(GitSource)).one().url == legacy
def test_get_masks_env_fallback_rows_with_embedded_token(
admin_client: TestClient, monkeypatch: pytest.MonkeyPatch
) -> None:
"""An env URL can embed a token too — the ENV VALUE itself is
untouched (the config is the operator's), only the response is
masked."""
monkeypatch.setattr(
git_sources_api,
"get_settings",
lambda: _settings(f"https://user:{TOKEN}@env.example.com/env.git"),
)
r = admin_client.get("/api/git-sources")
assert r.status_code == 200
assert TOKEN not in r.text
assert r.json()["from_env"] is True
assert r.json()["sources"][0]["url"] == "https://env.example.com/env.git"
def test_get_clean_urls_are_byte_identical(admin_client: TestClient, db: Session) -> None:
"""The phase-50/35 contract through the mask: credential-free
stored URLs (including ``git@`` and local paths) surface verbatim."""
urls = [
"https://github.com/owner/clean.git",
"git@github.com:owner/scp.git",
"ssh://git@example.com/repo.git",
]
base = datetime.now(UTC) - timedelta(hours=1)
# Distinct added_at: same-timestamp rows order by random uuid4 id.
db.add_all(
GitSource(url=u, added_at=base + timedelta(minutes=i)) for i, u in enumerate(urls)
)
db.commit()
body = admin_client.get("/api/git-sources").json()
assert [s["url"] for s in body["sources"]] == urls
# --- token: the sync clone URL (phase 121, task 02) -------------------------
@pytest.fixture()
def sync_admin_client() -> Iterator[TestClient]:
"""A context-managed, logged-in client — one app event loop across
requests (the sync background task must survive between the POST
and the polls; the test_sync_api.py pattern)."""
with TestClient(fastapi_app) as client:
r = client.post("/api/login", json={"password": "test-admin-password"})
assert r.status_code == 204
yield client
def _stub_sync_stack(
monkeypatch: pytest.MonkeyPatch, tmp_path: Path
) -> list[tuple[str, Path]]:
"""The sync pipeline minus the real git/import/LLM layers (the
test_sync_api.py pattern): a fresh settings dir, an empty env list
(the DB row wins), a no-op model probe, a zero-count import
(change-gated steps skipped), and a recording clone.
Returns the clone call list."""
monkeypatch.setattr(
sync_api,
"get_settings",
lambda: Settings(
_env_file=None, # pyright: ignore[reportCallIssue]
sources_dir=str(tmp_path / "sources"),
),
)
monkeypatch.setattr(
git_sources_resolver,
"get_settings",
lambda: Settings(_env_file=None, git_sources=""), # pyright: ignore[reportCallIssue]
)
async def fake_check_models(llm: object) -> None:
return None
monkeypatch.setattr(sync_api, "check_models", fake_check_models)
clone_calls: list[tuple[str, Path]] = []
def fake_clone_or_pull(url: str, dest: Path | str) -> Path:
dest = Path(dest)
dest.mkdir(parents=True, exist_ok=True)
clone_calls.append((url, dest))
return dest
monkeypatch.setattr(sync_api, "clone_or_pull", fake_clone_or_pull)
async def fake_import_sources(*args: object, **kwargs: object) -> ImportSummary:
return ImportSummary() # all-zero: the change-gated steps skip
monkeypatch.setattr(sync_api, "import_sources", fake_import_sources)
# The unchanged-walk gap probe: no candidates → no generator call
# (hermetic — the real probe reads the global KB state, and the
# generator would burn real lite calls).
monkeypatch.setattr(sync_api, "missing_folder_summaries", lambda db: [])
return clone_calls
def _poll_sync(client: TestClient, want: str, timeout: float = 5.0) -> dict:
deadline = time.monotonic() + timeout
while time.monotonic() < deadline:
body = client.get("/api/sync/status").json()
if body["state"] == want:
return body
assert body["state"] == "running", body
time.sleep(0.02)
raise AssertionError(f"sync did not reach {want!r} in {timeout}s")
def test_sync_token_row_clones_with_injected_url_and_bare_checkout(
sync_admin_client: TestClient, db: Session, monkeypatch: pytest.MonkeyPatch, tmp_path: Path
) -> None:
"""A token row's sync: ``clone_or_pull`` receives the injected
``https://x-access-token:<token>@…`` URL, the checkout directory is
derived from the BARE URL (credential-free), and no token leaks
into the status surface (raw text)."""
db.add(GitSource(url="https://github.com/owner/priv.git", token=TOKEN))
db.commit()
clone_calls = _stub_sync_stack(monkeypatch, tmp_path)
assert sync_admin_client.post("/api/sync").status_code == 202
body = _poll_sync(sync_admin_client, "success")
assert body["error"] is None
assert TOKEN not in str(body)
assert len(clone_calls) == 1
clone_url, dest = clone_calls[0]
assert clone_url == f"https://x-access-token:{TOKEN}@github.com/owner/priv.git"
# The checkout name is the bare repo name — no userinfo, no token.
assert dest.name == "priv"
assert dest == tmp_path / "sources" / "priv"
assert TOKEN not in str(dest)
def test_sync_legacy_row_clones_with_original_stored_url(
sync_admin_client: TestClient, db: Session, monkeypatch: pytest.MonkeyPatch, tmp_path: Path
) -> None:
"""Completion criterion: a pre-phase row (credential embedded in
the stored URL, token NULL) produces the ORIGINAL stored URL at
clone time — the credential keeps working — and the checkout name
is still the credential-free repo name (repo_name on the bare-URL
semantics: the basename after the last / of the stored URL)."""
stored = f"https://user:{TOKEN}@github.com/owner/priv.git"
db.add(GitSource(url=stored))
db.commit()
clone_calls = _stub_sync_stack(monkeypatch, tmp_path)
assert sync_admin_client.post("/api/sync").status_code == 202
body = _poll_sync(sync_admin_client, "success")
assert body["error"] is None
assert len(clone_calls) == 1
clone_url, dest = clone_calls[0]
assert clone_url == stored # the original stored URL, verbatim
assert dest.name == "priv"
def test_sync_public_row_clones_with_verbatim_url(
sync_admin_client: TestClient, db: Session, monkeypatch: pytest.MonkeyPatch, tmp_path: Path
) -> None:
"""Public repos behave byte-identically to pre-phase: a NULL-token
row's clone URL is the stored URL itself (no injection)."""
url = "https://github.com/owner/public.git"
db.add(GitSource(url=url))
db.commit()
clone_calls = _stub_sync_stack(monkeypatch, tmp_path)
assert sync_admin_client.post("/api/sync").status_code == 202
assert _poll_sync(sync_admin_client, "success")["error"] is None
assert clone_calls == [(url, tmp_path / "sources" / "public")]
tests/integration/test_migration_0021.py
+280
View File
@@ -0,0 +1,280 @@
"""Integration: migration 0021 (git_sources.token) schema contract
(phase 121, task 01).
Drives the **real Alembic engine** against the live dev database
(``podman compose up -d db``), mirroring the house pattern of
``test_migration_0020.py`` (information_schema assertions on the state
the migration must leave). The tests target the 0020 → 0021 step
explicitly so later migrations cannot break the pins:
* upgrade 0020 → 0021 → ``token`` exists with the full contract —
TEXT, NULLABLE, no server default — while the 0020 ``git_sources``
schema (``url`` NOT NULL + the unique index, ``kind``, ``path``,
``ignore_paths``, ``include_hidden``, ``added_at``) survives;
* a ``git_sources`` row inserted while the DB is at 0020 backfills
``token`` to NULL (a pre-phase-121 row is a public repo — or a
legacy embedded-token row whose credential lives in ``url``);
* the ORM contract agrees: a freshly inserted ``GitSource`` with an
explicit ``token`` round-trips it through a fresh session, and a row
without one reads ``token is None``;
* downgrade to 0020 → the column is GONE (A13) while the row + its
``url`` survive; upgrade back to 0021 → the column is back
(round-trip).
The ``alembic`` fixture guarantees the DB ends at head even if a test
fails or the process is interrupted.
"""
from __future__ import annotations
import uuid
from collections.abc import Iterator
from typing import Any
import pytest
from alembic.config import Config
from sqlalchemy import text
from sqlalchemy.orm import Session
from alembic import command
from app.db import SessionLocal, db_available
from app.models import GitSource
URL_BARE = "https://github.com/mig0021/bare.git"
URL_TOKENED = "https://github.com/mig0021/tokened.git"
TOKEN = "ghp_mig0021secret"
@pytest.fixture()
def alembic(db: Session) -> Iterator[Config]:
"""Real Alembic config bound to the dev DB (URL from app settings).
Starts at head (repairs an interrupted earlier run); teardown
upgrades to head no matter what happened, so the dev DB is never
left below head.
"""
if not db_available():
pytest.skip("Postgres not reachable — run `podman compose up -d db` first")
cfg = Config() # no alembic.ini file — env.py gets the URL from app config
cfg.set_main_option("script_location", "alembic")
command.upgrade(cfg, "head")
try:
yield cfg
finally:
# Release the test session's open transaction BEFORE the repair
# DDL: an idle-in-transaction SELECT holds an ACCESS SHARE lock
# on ``git_sources``, which would deadlock the repair's
# ``ALTER TABLE`` (0021) forever.
db.rollback()
command.upgrade(cfg, "head")
def _version(db: Session) -> str | None:
return db.execute(text("SELECT version_num FROM alembic_version")).scalar()
def _column(db: Session, column: str) -> tuple[Any, ...] | None:
"""(data_type, is_nullable, column_default) for one git_sources
column."""
row = db.execute(
text(
"SELECT data_type, is_nullable, column_default"
" FROM information_schema.columns"
" WHERE table_name = 'git_sources' AND column_name = :c"
),
{"c": column},
).fetchone()
return tuple(row) if row is not None else None
def _insert_sql(db: Session, url: str) -> uuid.UUID:
"""Insert one git_sources row with the PRE-0021 column set (the
0020 shape — the token column, when present, is omitted so a NULL
backfill is what the row reads)."""
row_id = uuid.uuid4()
db.execute(
text(
"INSERT INTO git_sources (id, url, kind, path, ignore_paths,"
" include_hidden)"
" VALUES (:id, :u, 'git', NULL, '[]', false)"
),
{"id": row_id, "u": url},
)
db.commit()
return row_id
def _delete(db: Session, row_id: uuid.UUID) -> None:
db.execute(text("DELETE FROM git_sources WHERE id = :id"), {"id": row_id})
db.commit()
def test_upgrade_to_0021_adds_token(db: Session, alembic: Config) -> None:
"""Upgrade 0020 → 0021: ``token`` exists with the full contract
(TEXT, NULLABLE, no server default — NULL = public/legacy row), is
ABSENT at 0020, a pre-0021 row backfills ``token`` to NULL, and an
omitted token on a new row stays NULL — while the 0020 table
contract (``url`` + unique index, ``kind``, ``ignore_paths``,
``include_hidden``, ``added_at``) survives."""
command.downgrade(alembic, "0020") # start from the pre-0021 state
assert _version(db) == "0020"
assert _column(db, "token") is None, "token must be absent at 0020"
pre_id = _insert_sql(db, URL_BARE) # the 0020 column set
try:
command.upgrade(alembic, "0021")
assert _version(db) == "0021", "alembic_version must be at 0021"
token = _column(db, "token")
assert token is not None, "git_sources.token is missing"
assert token[0] == "text", "token must be TEXT"
assert token[1] == "YES", "token must be NULLABLE"
assert token[2] is None, (
"token must carry NO server default — NULL is the"
" public/legacy value"
)
# The pre-0021 row backfilled to NULL (a public repo, or a
# legacy embedded-token row whose credential lives in url).
row = db.execute(
text("SELECT url, token FROM git_sources WHERE id = :id"),
{"id": pre_id},
).fetchone()
assert row is not None and row[0] == URL_BARE, (
"the pre-0021 row must survive the upgrade"
)
assert row[1] is None, "the backfilled token must be NULL"
# A row written without the column stays NULL (no default to
# fill it in — the Python-side default is None, same value).
new_id = _insert_sql(db, URL_TOKENED)
try:
tokened = db.execute(
text("SELECT token FROM git_sources WHERE id = :id"),
{"id": new_id},
).scalar()
assert tokened is None, "an omitted token must stay NULL"
finally:
_delete(db, new_id)
# The 0020 schema survives the additive upgrade.
url = _column(db, "url")
assert url is not None and url[0] == "text" and url[1] == "NO", (
"git_sources.url (0006) must keep its 0020 contract"
)
kind = _column(db, "kind")
assert kind is not None and kind[0] == "text" and kind[1] == "NO", (
"git_sources.kind (0007) must survive the upgrade"
)
ignore = _column(db, "ignore_paths")
assert ignore is not None and ignore[0] == "jsonb" and ignore[1] == "NO", (
"git_sources.ignore_paths (0013) must survive the upgrade"
)
hidden = _column(db, "include_hidden")
assert hidden is not None and hidden[0] == "boolean" and hidden[1] == "NO", (
"git_sources.include_hidden (0019) must survive the upgrade"
)
added = _column(db, "added_at")
assert added is not None and added[0] == "timestamp with time zone", (
"git_sources.added_at (0006) must survive the upgrade"
)
assert added[1] == "NO" and "now()" in str(added[2]), (
"git_sources.added_at must keep its `now()` server default"
)
index = db.execute(
text(
"SELECT indexname FROM pg_indexes"
" WHERE tablename = 'git_sources'"
" AND indexname = 'uq_git_sources_url'"
)
).scalar()
assert index is not None, (
"the uq_git_sources_url unique index must survive the upgrade"
)
finally:
_delete(db, pre_id)
def test_orm_token_round_trips(db: Session, alembic: Config) -> None:
"""The ORM contract agrees with the column contract: a freshly
inserted ``GitSource`` with an explicit ``token`` round-trips the
credential through a FRESH session, and a row without one reads
``token is None`` (the NULL public/legacy state)."""
command.upgrade(alembic, "head")
row_tokened = GitSource(url=URL_TOKENED, kind="git", token=TOKEN)
row_bare = GitSource(url=URL_BARE, kind="git")
db.add(row_tokened)
db.add(row_bare)
db.commit()
try:
with SessionLocal() as fresh:
reloaded_tokened = fresh.get(GitSource, row_tokened.id)
assert reloaded_tokened is not None, "the tokened row must be readable"
assert reloaded_tokened.token == TOKEN, (
"the explicit token must round-trip through the DB"
)
reloaded_bare = fresh.get(GitSource, row_bare.id)
assert reloaded_bare is not None, "the bare row must be readable"
assert reloaded_bare.token is None, (
"a row without a token must read token is None"
)
finally:
_delete(db, row_tokened.id)
_delete(db, row_bare.id)
def test_downgrade_to_0020_drops_the_column(db: Session, alembic: Config) -> None:
"""Downgrade 0021 → 0020: the token column is gone (A13 — fully
reversible) while the row + its ``url`` survive, and the rest of
the 0020 table contract (``url`` unique index, ``kind``,
``added_at``) is intact."""
command.upgrade(alembic, "head")
row = GitSource(url=URL_TOKENED, kind="git", token=TOKEN)
db.add(row)
db.commit()
try:
command.downgrade(alembic, "0020")
assert _version(db) == "0020"
assert _column(db, "token") is None, "token must be dropped"
surviving = db.execute(
text(
"SELECT url, kind, ignore_paths, include_hidden, added_at"
" FROM git_sources WHERE id = :id"
),
{"id": row.id},
).fetchone()
assert surviving is not None and surviving[0] == URL_TOKENED, (
"the row must survive the column drop"
)
assert surviving[1] == "git" and surviving[2] == [] and surviving[3] is False, (
"kind + ignore_paths + include_hidden must survive the drop"
)
assert surviving[4] is not None, "added_at must survive the drop"
index = db.execute(
text(
"SELECT indexname FROM pg_indexes"
" WHERE tablename = 'git_sources'"
" AND indexname = 'uq_git_sources_url'"
)
).scalar()
assert index is not None, "the unique index must survive the downgrade"
finally:
_delete(db, row.id)
# Repair: the fixture teardown re-upgrades to head.
def test_upgrade_round_trip_restores_the_column(db: Session, alembic: Config) -> None:
"""Downgrade to 0020, then upgrade back to 0021: ``token`` is back
with the full contract (TEXT, NULLABLE, no server default)."""
command.downgrade(alembic, "0020")
command.upgrade(alembic, "0021")
assert _version(db) == "0021", "round-trip upgrade must land at 0021"
token = _column(db, "token")
assert token is not None, "git_sources.token must be back"
assert token[0] == "text", "token must be TEXT after the round-trip"
assert token[1] == "YES", "token must be NULLABLE after the round-trip"
assert token[2] is None, (
"token must still carry NO server default after the round-trip"
)
tests/unit/test_git_source_token.py
+641
View File
@@ -0,0 +1,641 @@
"""Unit: the git-source token schema surface (phase 121, task 01).
Task 01 lands the STORAGE only: the write-side input shapes can carry
the masked credential, and the output shapes structurally cannot.
* ``GitSourceIn.token`` / ``GitSourcePatchIn.token`` — optional
``str | None`` (absent/None = no credential / no change), trimmed
*before* the max-500 length constraint runs (the ``_trim_url``
precedent), ``None`` passing through untouched;
* ``GitSourcePatchIn.token`` — the tri-state documented contract:
absent/None = no change, non-empty = replace, empty string = clear;
* ``GitSourceOut`` / ``GitSourceRow`` — NO ``token`` field (LOCKED A2):
the credential never reaches the UI or any API output, and
``extra="forbid"`` makes the omission structural — constructing an
output model with a ``token`` key (kwarg OR dict) raises, so a
regression that tries to echo the credential back cannot even build
the shape (task 04 finalizes this file with the E2E-level pins).
Task 02 lands the CLONE + SANITIZATION mechanics (pure helpers, no
DB):
* ``sanitize_url`` — strips the userinfo of ``https?://`` URLs
(``user:pass@`` and the username-as-token form), leaves ``ssh://`` /
``git@`` / local paths untouched, is idempotent, and is
byte-identical for credential-free URLs (anchored regex, never a
URL parser re-serialization);
* ``clone_url_for`` — NULL/falsy token → the bare stored URL verbatim
(public + legacy rows clone exactly as pre-phase), https? row with
a token → ``https://x-access-token:<token>@…`` (any existing
userinfo replaced by the column credential), non-https row with a
token → the URL unchanged + a warning log (no crash);
* ``normalize_credential`` — embedded userinfo moves to the token
column (password part of ``user:pass``; the whole run for the
username-as-token form), an explicit token (even "") wins (LOCKED
A6), clean URLs + ssh/``git@``/local paths come back untouched.
"""
from __future__ import annotations
import re
import uuid
from pathlib import Path
import pytest
from pydantic import ValidationError
from app.models import GitSource
from app.rag.git_sources import clone_url_for, normalize_credential, sanitize_url
from app.schemas import GitSourceIn, GitSourceOut, GitSourcePatchIn, GitSourceRow
URL = "https://github.com/owner/repo.git"
# ---------------------------------------------------------------------------
# GitSourceIn — write side: accepts + trims the token
# ---------------------------------------------------------------------------
def test_git_source_in_token_defaults_to_none() -> None:
"""Absent token = no credential (public repo) — the pre-phase-121
body shape still validates unchanged."""
payload = GitSourceIn(url=URL)
assert payload.token is None
def test_git_source_in_token_accepted() -> None:
payload = GitSourceIn(url=URL, token="ghp_secret123")
assert payload.token == "ghp_secret123"
def test_git_source_in_token_trimmed_before_length_constraints() -> None:
"""The ``_trim_url`` precedent: surrounding whitespace is stripped
before the max-500 constraint runs."""
payload = GitSourceIn(url=URL, token=" ghp_secret123 ")
assert payload.token == "ghp_secret123"
def test_git_source_in_token_whitespace_only_becomes_empty() -> None:
"""Trimmed to ``""`` — valid (create-time has no min_length): the
caller sent a blank masked field, i.e. no credential."""
payload = GitSourceIn(url=URL, token=" ")
assert payload.token == ""
def test_git_source_in_token_at_cap_validates() -> None:
token = "x" * 500
payload = GitSourceIn(url=URL, token=token)
assert payload.token == token
def test_git_source_in_token_one_over_cap_rejects() -> None:
with pytest.raises(ValidationError) as exc:
GitSourceIn(url=URL, token="x" * 501)
assert exc.value.errors()[0]["loc"] == ("token",)
def test_git_source_in_pre_phase_fields_still_validate() -> None:
"""The pre-phase-121 body (url + ignore_paths + include_hidden) is
unchanged — the new field is purely additive."""
payload = GitSourceIn(url=URL, ignore_paths=["docs/private"], include_hidden=True)
assert payload.url == URL
assert payload.ignore_paths == ["docs/private"]
assert payload.include_hidden is True
assert payload.token is None
# ---------------------------------------------------------------------------
# GitSourcePatchIn — tri-state: absent/None no change, non-empty replace,
# empty string clear
# ---------------------------------------------------------------------------
def test_git_source_patch_in_token_defaults_to_none() -> None:
"""Absent/None = NO CHANGE — the row's stored credential survives
an edit that does not touch the masked field (the edit modal sends
it blank → the router omits the key → None here)."""
payload = GitSourcePatchIn()
assert payload.token is None
def test_git_source_patch_in_token_replace_value() -> None:
payload = GitSourcePatchIn(token="new-secret")
assert payload.token == "new-secret"
def test_git_source_patch_in_token_trimmed() -> None:
payload = GitSourcePatchIn(token=" new-secret ")
assert payload.token == "new-secret"
def test_git_source_patch_in_token_empty_string_clears() -> None:
"""Empty string = CLEAR (the UI offers replace; clear exists for
API completeness)."""
payload = GitSourcePatchIn(token="")
assert payload.token == ""
def test_git_source_patch_in_token_whitespace_only_clears() -> None:
"""Whitespace-only trims to the clear value — a pasted space in the
masked field is a clear, not a five-character credential."""
payload = GitSourcePatchIn(token=" ")
assert payload.token == ""
def test_git_source_patch_in_token_at_cap_validates() -> None:
token = "x" * 500
payload = GitSourcePatchIn(token=token)
assert payload.token == token
def test_git_source_patch_in_token_one_over_cap_rejects() -> None:
with pytest.raises(ValidationError) as exc:
GitSourcePatchIn(token="x" * 501)
assert exc.value.errors()[0]["loc"] == ("token",)
def test_git_source_patch_in_other_fields_unaffected() -> None:
"""ignore_paths / include_hidden keep their independent optionality
— the new field adds a third state, not a coupling."""
payload = GitSourcePatchIn(ignore_paths=["a"], include_hidden=True)
assert payload.ignore_paths == ["a"]
assert payload.include_hidden is True
assert payload.token is None
# ---------------------------------------------------------------------------
# GitSourceOut / GitSourceRow — NO token field, ever (LOCKED A2)
# ---------------------------------------------------------------------------
def _out_kwargs() -> dict:
return {
"id": uuid.uuid4(),
"url": URL,
"added_at": None,
"ignore_paths": [],
"include_hidden": False,
}
def _row_kwargs() -> dict:
return {
"id": uuid.uuid4(),
"kind": "git",
"url": URL,
"path": None,
"added_at": None,
"ignore_paths": [],
"include_hidden": False,
}
def test_output_models_have_no_token_field() -> None:
"""The omission is a documented contract, not an accident: neither
response shape declares a token field at all."""
assert "token" not in GitSourceOut.model_fields
assert "token" not in GitSourceRow.model_fields
def test_git_source_out_rejects_token_kwarg() -> None:
"""``extra="forbid"`` — a regression that tries to echo the stored
credential back cannot even build the shape. (The kwarg is a
deliberate type error — pyright statically knows the shape has no
token parameter; that is the contract under test.)"""
with pytest.raises(ValidationError) as exc:
GitSourceOut(token="ghp_must_never_leak", **_out_kwargs()) # type: ignore[reportCallIssue]
assert ("token",) in [tuple(e["loc"]) for e in exc.value.errors()]
def test_git_source_out_rejects_token_key() -> None:
with pytest.raises(ValidationError) as exc:
GitSourceOut.model_validate({**_out_kwargs(), "token": "ghp_must_never_leak"})
assert ("token",) in [tuple(e["loc"]) for e in exc.value.errors()]
def test_git_source_row_rejects_token_kwarg() -> None:
with pytest.raises(ValidationError) as exc:
# Same deliberate type error as the GitSourceOut pin above — the
# shape has no token parameter (pyright knows; runtime forbids).
GitSourceRow(token="ghp_must_never_leak", **_row_kwargs()) # type: ignore[reportCallIssue]
assert ("token",) in [tuple(e["loc"]) for e in exc.value.errors()]
def test_git_source_row_rejects_token_key() -> None:
with pytest.raises(ValidationError) as exc:
GitSourceRow.model_validate({**_row_kwargs(), "token": "ghp_must_never_leak"})
assert ("token",) in [tuple(e["loc"]) for e in exc.value.errors()]
def test_output_models_still_build_with_declared_fields() -> None:
"""The forbid boundary rejects the credential, not the row: the
declared-field construction every endpoint uses still validates."""
out = GitSourceOut(**_out_kwargs())
assert out.url == URL
row = GitSourceRow(**_row_kwargs())
assert row.url == URL and row.kind == "git"
# ---------------------------------------------------------------------------
# sanitize_url (task 02) — the OUTPUT mask: userinfo stripped from
# https? URLs only, byte-identical for everything else
# ---------------------------------------------------------------------------
def test_sanitize_strips_user_pass_userinfo() -> None:
"""The TODO L5 shape — the embedded credential is gone, the bare
host/path survive character for character."""
assert (
sanitize_url("https://myuser:ghp_xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx@github.com/owner/private-repo.git")
== "https://github.com/owner/private-repo.git"
)
def test_sanitize_strips_username_as_token_form() -> None:
"""The documented GitHub shape (no colon in the userinfo): the
whole run is stripped too."""
assert sanitize_url("https://ghp_onlytoken@github.com/owner/repo.git") == (
"https://github.com/owner/repo.git"
)
def test_sanitize_handles_http_scheme() -> None:
assert sanitize_url("http://user:pass@git.local/repo.git") == "http://git.local/repo.git"
def test_sanitize_leaves_ssh_scp_and_local_untouched() -> None:
"""``ssh://``, scp-style ``git@``, and local paths carry no
userinfo (or are not secrets in this shape) — untouched."""
for url in (
"ssh://git@example.com/repo.git",
"git@github.com:owner/repo.git",
"/home/owner/bor-sources/repo",
"~/Homelab",
):
assert sanitize_url(url) == url, url
def test_sanitize_is_byte_identical_for_credential_free_urls() -> None:
"""The phase-50/35 contract: stored URLs surface verbatim when they
carry no credential — an anchored regex replace, never a URL
parser re-serialization (a parser would rewrite the path)."""
url = "https://github.com/owner/repo.git?ref=main#readme"
assert sanitize_url(url) == url
def test_sanitize_at_in_path_is_not_userinfo() -> None:
"""A ``@`` inside the *path* is not userinfo — the anchor requires
the run to come right after the scheme."""
url = "https://github.com/owner/repo@v1/blob/x"
assert sanitize_url(url) == url
def test_sanitize_is_idempotent() -> None:
embedded = "https://user:pass@github.com/owner/repo.git"
once = sanitize_url(embedded)
assert sanitize_url(once) == once
def test_sanitize_empty_and_scheme_only_urls() -> None:
assert sanitize_url("") == ""
assert sanitize_url("https://") == "https://"
# ---------------------------------------------------------------------------
# clone_url_for (task 02) — the CLONE-time credential injection
# ---------------------------------------------------------------------------
def test_clone_url_null_token_returns_stored_url_verbatim() -> None:
"""Public repos (and local rows) clone byte-identically to
pre-phase-121 — no injection, no logging."""
row = GitSource(url="https://github.com/owner/public.git", token=None)
assert clone_url_for(row) == "https://github.com/owner/public.git"
def test_clone_url_empty_token_is_falsy_no_injection() -> None:
row = GitSource(url="https://github.com/owner/public.git", token="")
assert clone_url_for(row) == "https://github.com/owner/public.git"
def test_clone_url_injects_x_access_token_for_https() -> None:
"""The task-02 assumption (step 4): ``x-access-token`` as the
userinfo username — GitHub-agnostic, reads as non-identifying."""
row = GitSource(url="https://github.com/owner/private.git", token="ghp_secret123")
assert (
clone_url_for(row) == "https://x-access-token:ghp_secret123@github.com/owner/private.git"
)
def test_clone_url_injects_for_http_scheme() -> None:
row = GitSource(url="http://git.local/repo.git", token="tok")
assert clone_url_for(row) == "http://x-access-token:tok@git.local/repo.git"
def test_clone_url_replaces_existing_userinfo_with_column_credential() -> None:
"""A legacy stored URL that still embeds userinfo (only reachable
via direct-DB writes now — the API normalizes at write time) gets
the column credential, not the embedded one."""
row = GitSource(url="https://user:oldpass@github.com/owner/repo.git", token="newpass")
assert clone_url_for(row) == "https://x-access-token:newpass@github.com/owner/repo.git"
def test_clone_url_legacy_row_clones_with_original_stored_url() -> None:
"""Completion criterion: a pre-phase row (credential embedded in
the stored URL, token NULL) produces the ORIGINAL stored URL at
clone time — the credential keeps working."""
stored = "https://user:ghp_secret@github.com/owner/repo.git"
row = GitSource(url=stored, token=None)
assert clone_url_for(row) == stored
@pytest.mark.parametrize(
("url", "token"),
[
("ssh://git@example.com/repo.git", "sekrit-value"),
("git@github.com:owner/repo.git", "sekrit-value"),
("/home/owner/bor-sources/repo", "sekrit-value"),
],
)
def test_clone_url_non_https_token_is_noop_with_warning(
caplog: pytest.LogCaptureFixture, url: str, token: str
) -> None:
"""A token cannot authenticate ssh/scp/local — the URL is returned
unchanged, a warning is logged (no crash), and the token VALUE
never leaks into the log line."""
row = GitSource(url=url, token=token)
with caplog.at_level("WARNING", logger="app.rag.git_sources"):
assert clone_url_for(row) == url
warnings = [r for r in caplog.records if r.levelname == "WARNING"]
assert len(warnings) == 1
assert "sekrit-value" not in warnings[0].getMessage()
# ---------------------------------------------------------------------------
# normalize_credential (task 02) — the WRITE-path normalizer
# ---------------------------------------------------------------------------
def test_normalize_moves_embedded_password_to_column() -> None:
"""The TODO L5 paste: ``user:pass@`` → bare URL + the PASSWORD part
as the token (the username is an identifier, not the credential —
the clone-time injection ``x-access-token:<password>@`` must
authenticate)."""
bare, effective = normalize_credential(
"https://myuser:ghp_secret123@github.com/owner/private-repo.git", None
)
assert bare == "https://github.com/owner/private-repo.git"
assert effective == "ghp_secret123"
def test_normalize_password_may_contain_colons() -> None:
"""The split is on the FIRST colon only — ``user:a:b`` moves
``a:b`` wholesale."""
bare, effective = normalize_credential("https://user:a:b@github.com/x/y.git", None)
assert bare == "https://github.com/x/y.git"
assert effective == "a:b"
def test_normalize_username_as_token_form_moves_whole_run() -> None:
"""The username-as-token form (no colon in the userinfo) — the
whole run is the credential."""
bare, effective = normalize_credential("https://ghp_onlytoken@github.com/x/y.git", None)
assert bare == "https://github.com/x/y.git"
assert effective == "ghp_onlytoken"
def test_normalize_explicit_token_wins_over_embedded() -> None:
"""LOCKED A6 — explicit beats embedded (the masked field and the
pasted URL disagree: the field is the intent)."""
bare, effective = normalize_credential(
"https://user:ghp_embedded@github.com/x/y.git", "ghp_explicit"
)
assert bare == "https://github.com/x/y.git"
assert effective == "ghp_explicit"
def test_normalize_explicit_empty_token_wins_and_clears() -> None:
"""An explicit "" (blank masked field) is a deliberate no-credential
declaration: it beats the embedded one and stores NULL (the caller
stores ``effective or None``)."""
bare, effective = normalize_credential("https://user:ghp_x@github.com/x/y.git", "")
assert bare == "https://github.com/x/y.git"
assert effective == ""
def test_normalize_clean_url_with_explicit_token_untouched() -> None:
"""The common UI path (bare URL + masked field): the URL is
byte-identical, the token passes through."""
bare, effective = normalize_credential("https://github.com/x/y.git", "ghp_t")
assert bare == "https://github.com/x/y.git"
assert effective == "ghp_t"
def test_normalize_clean_url_without_token_untouched() -> None:
assert normalize_credential("https://github.com/x/y.git", None) == (
"https://github.com/x/y.git",
None,
)
def test_normalize_non_https_urls_untouched() -> None:
"""ssh/scp/local carry no userinfo — returned untouched, token
(whatever it is) passing through for the caller's tri-state."""
for url in ("ssh://git@example.com/repo.git", "git@github.com:x/y.git", "/local/dir"):
assert normalize_credential(url, None) == (url, None)
assert normalize_credential(url, "t") == (url, "t")
def test_normalize_idempotent_on_bare_url() -> None:
bare, _ = normalize_credential("https://user:pass@github.com/x/y.git", None)
again, effective = normalize_credential(bare, "tok")
assert again == bare
assert effective == "tok"
# ---------------------------------------------------------------------------
# Frontend source pins (task 03) — the masked token field: add form +
# per-row editor; the display sites stay on the server-sanitized s.url
# ---------------------------------------------------------------------------
FRONTEND = Path(__file__).resolve().parents[2] / "frontend"
SHELL_HTML = FRONTEND / "index.html"
GIT_SOURCES_JS = FRONTEND / "assets" / "git-sources.js"
STYLES_CSS = FRONTEND / "assets" / "styles.css"
ADD_TOKEN_PLACEHOLDER = "ghp_… or another PAT"
EDIT_TOKEN_PLACEHOLDER = "leave blank to keep the current token"
ADD_BODY_EXPR = "(url, token) => ({ url, ...(token ? { token } : {}) })"
PATCH_BODY_EXPR = "JSON.stringify({ ignore_paths: lines, ...(token ? { token } : {}) })"
def _read(path: Path) -> str:
return path.read_text(encoding="utf-8")
def _js_text() -> str:
return _read(GIT_SOURCES_JS)
def _fn(js: str, name: str) -> str:
"""The source of a (possibly async) top-level function via
balanced-brace counting (the test_source_ignore_paths.py helper)."""
for prefix in ("async function ", "function "):
start = js.find(f"{prefix}{name}(")
if start != -1:
depth = 0
for i in range(js.find("{", start), len(js)):
if js[i] == "{":
depth += 1
elif js[i] == "}":
depth -= 1
if depth == 0:
return js[start : i + 1]
raise AssertionError(f"unbalanced braces in {name}()")
raise AssertionError(f"{name}() must exist in git-sources.js")
def _input_tag(html: str, id_attr: str) -> str:
m = re.search(rf"<input[^>]*id=\"{id_attr}\"[^>]*>", html, re.S)
assert m, f"missing <input id={id_attr}> in the shell"
return m.group(0)
def test_add_form_token_field_is_masked_and_optional() -> None:
"""The add form's `#git-source-token` (task 03 step 1, LOCKED A2):
``type="password"`` (masked — the credential is never a visible-text
field in the DOM), ``autocomplete="off"`` (a PAT is not a site
credential — no browser save offer), the 500-char cap mirrored, the
placeholder, INSIDE `#git-source-form` before the submit button,
and a visible `<label for=…>` (WCAG — never aria-label-only)
carrying the "optional" hint span."""
html = _read(SHELL_HTML)
tag = _input_tag(html, "git-source-token")
assert 'type="password"' in tag, "the token is masked (type=password)"
assert 'autocomplete="off"' in tag, "no browser save offer (ASSUMPTION, task 03)"
assert 'maxlength="500"' in tag, "the schema cap, mirrored in the form"
assert ADD_TOKEN_PLACEHOLDER in tag
form_i = html.find('id="git-source-form"')
btn_i = html.find('id="git-source-add"')
tok_i = html.find('id="git-source-token"')
assert -1 < form_i < tok_i < btn_i, "the field sits in the add form, before the submit"
label = re.search(r"<label[^>]*for=\"git-source-token\"[^>]*>(.*?)</label>", html, re.S)
assert label, "a visible <label for=…> (never aria-label-only)"
assert "Token" in label.group(1)
hint = re.search(r"<span[^>]*class=\"field-hint\"[^>]*>(.*?)</span>", label.group(1))
assert hint and "optional" in hint.group(1), "the visible 'optional' hint span"
def test_add_form_submit_omits_a_blank_token() -> None:
"""The submit body is `(url, token) => ({ url, ...(token ?
{ token } : {}) })` — a BLANK token omits the key (None = no
credential); wireAddForm reads the masked field and passes it to
the body callback, and 201 clears BOTH inputs (the credential is
stored — write-only)."""
js = _js_text()
assert f"body: {ADD_BODY_EXPR}" in js, "the submit body omits the key when blank"
wire = _fn(js, "wireAddForm")
read_i = wire.find('tokenInput ? tokenInput.value.trim() : ""')
call_i = wire.find("opts.body(value, token)", read_i)
assert -1 < read_i < call_i, "the token is read from the field and passed to the body"
clear_url_i = wire.find('input.value = ""')
clear_tok_i = wire.find("tokenInput.value = \"\"", clear_url_i)
assert -1 < clear_url_i < clear_tok_i, "201: both inputs clear"
def test_editor_token_field_is_masked_and_blanks_to_keep() -> None:
"""The per-row editor's `#ignore-editor-token` (task 03 step 2):
``type="password"`` + ``autocomplete="off"`` + the "leave blank to
keep the current token" placeholder, INSIDE `#ignore-editor-dialog`
(after the textarea, before the error line), with a visible label.
The JS always opens the field BLANK (the API has no token field to
prefill from — LOCKED A2) and resets it on close."""
html = _read(SHELL_HTML)
tag = _input_tag(html, "ignore-editor-token")
assert 'type="password"' in tag
assert 'autocomplete="off"' in tag
assert f'placeholder="{EDIT_TOKEN_PLACEHOLDER}"' in tag
dialog_i = html.find('id="ignore-editor-dialog"')
ta_i = html.find('id="ignore-editor-textarea"')
tok_i = html.find('id="ignore-editor-token"')
err_i = html.find('id="ignore-editor-error"')
assert -1 < dialog_i < ta_i < tok_i < err_i, ("the field sits in the editor, after the box")
label = re.search(r"<label[^>]*for=\"ignore-editor-token\"[^>]*>(.*?)</label>", html, re.S)
assert label and "Token" in label.group(1), "a visible label (WCAG)"
js = _js_text()
open_body = _fn(js, "openIgnoreEditor")
close_body = _fn(js, "closeIgnoreEditor")
assert 'ignoreTokenInput.value = ""' in open_body, ("opens BLANK — never prefilled")
assert 'ignoreTokenInput.value = ""' in close_body, ("resets on close")
def test_editor_patch_omits_a_blank_token() -> None:
"""saveIgnorePaths: the token is read AFTER the line parse and the
PATCH body is `{ ignore_paths: lines, ...(token ? { token } : {}) }`
— a BLANK token omits the key (the task-01 tri-state: absent = no
change, the row's stored credential is kept)."""
js = _js_text()
save = _fn(js, "saveIgnorePaths")
drop_i = save.find(".filter(Boolean)")
read_i = save.find('ignoreTokenInput ? ignoreTokenInput.value.trim() : ""', drop_i)
body_i = save.find(PATCH_BODY_EXPR, read_i)
assert -1 < drop_i < read_i < body_i, ("blank token → key omitted (tri-state no-change)")
def test_display_sites_render_the_server_bare_url_only() -> None:
"""Task 03 step 3: every display site keeps rendering `s.url`
UNCHANGED (the server now returns bare URLs — sanitize_url, phase
121), the one-line source note pins that contract, and NO display
site ever reads a token off the row (the response has no token
field — and even if one did, the UI must never re-embed it)."""
js = _js_text()
make = _fn(js, "makeRow")
assert "const value = isLocal ? (s.path ?? s.url) : s.url;" in make, (
"the display value is the server row's url/path — UNCHANGED"
)
assert "sanitized server-side" in make, ("the one-line phase-121 note at the display site")
assert "never re-embed a credential" in make
assert "s.token" not in js, "the UI never re-embeds a credential (LOCKED A2)"
def test_module_docstring_carries_the_phase_121_token_contract() -> None:
"""The git-sources.js module docstring gained the phase-121 bullet:
the masked add field (type=password, autocomplete=off, the "optional
— private repos" hint), the blank-omits-key submit body, the
editor's "leave blank to keep the current token" mirror, and the
display-sites-unchanged (sanitize_url) contract."""
doc = _js_text().split("*/", 2)[0] # the module docstring (first block)
for frag in (
"Phase 121 (task 03)",
"#git-source-token",
"type=password",
"optional — private repos",
ADD_BODY_EXPR,
"leave blank to keep the current token",
"sanitize_url",
):
assert frag in doc, f"the module docstring lost: {frag!r}"
def test_token_field_css_rules_are_house_styled() -> None:
"""styles.css (task 03 step 2 — no new CSS beyond the theme's
input treatment): the add form's token input shares the URL
input's box (grouped selector — mono, >=44px floor); the
`.field-hint` rule is muted (ink-soft — 5.1:1 on the label's
surface, AA) and small; the editor's token field is full panel
width at the 44px floor; the mobile media query groups it in the
same min-width:0 rule as the URL input."""
css = _read(STYLES_CSS)
grouped = css.find("#git-source-url,\n#git-source-token {")
assert grouped != -1, "#git-source-token shares #git-source-url's box"
rule = css[grouped : css.find("}", grouped)]
assert "min-height: 44px" in rule and "var(--mono)" in rule
hint = css.find(".field-hint {")
assert hint != -1, "the .field-hint rule"
hint_rule = css[hint : css.find("}", hint)]
assert "var(--ink-soft)" in hint_rule and "font-size: 0.8rem" in hint_rule
tok = css.find(".ignore-editor-token {")
assert tok != -1, "the editor's token field rule"
tok_rule = css[tok : css.find("}", tok)]
assert "width: 100%" in tok_rule and "min-height: 44px" in tok_rule
mobile = css.find("#git-source-url,\n #git-source-token,")
assert mobile != -1, "the mobile squeeze groups the token input"
tests/unit/test_source_ignore_paths.py
+15 -6
View File
@@ -68,6 +68,7 @@ DIALOG_IDS = (
"ignore-editor-source",
"ignore-editor-copy",
"ignore-editor-textarea",
"ignore-editor-token", # phase 121 (task 03): the masked token field
"ignore-editor-error",
"ignore-editor-cancel",
"ignore-editor-save",
@@ -359,7 +360,10 @@ def test_save_runs_the_inflight_never_stale_lifecycle() -> None:
A4). The §7.4 in-flight state precedes the PATCH: both buttons
disable + the save relabels "Saving…" — one
``PATCH /api/git-sources/{id}`` with the lines as the whole
body list (A5 replace). 200 → close (focus return) →
body list (A5 replace). Phase 121 (task 03): the masked token is
read AFTER the line parse and included in the body ONLY when
non-blank (the tri-state: blank → key omitted → no change, the
row's stored credential is kept). 200 → close (focus return) →
loadSources (the count tag lands) → announce (the update
confirmation is the LAST announcement). Non-2xx: the in-dialog
role=alert line (apiDetail, 422 shape-aware), the dialog STAYS
@@ -371,18 +375,23 @@ def test_save_runs_the_inflight_never_stale_lifecycle() -> None:
parse_i = body.find('.split("\\n")')
trim_i = body.find(".map((l) => l.trim())", parse_i)
drop_i = body.find(".filter(Boolean)", trim_i)
inflight_i = body.find("ignoreInFlight = true", drop_i)
# Phase 121 (task 03): the token is read between the line parse
# and the in-flight flag (blank = the key is omitted).
token_i = body.find('ignoreTokenInput ? ignoreTokenInput.value.trim() : ""', drop_i)
inflight_i = body.find("ignoreInFlight = true", token_i)
dis_c = body.find("ignoreCancelBtn.disabled = true", inflight_i)
dis_s = body.find("ignoreSaveBtn.disabled = true", inflight_i)
label_i = body.find(f'"{SAVING_LABEL}"', dis_s)
fetch_i = body.find("`/api/git-sources/${", label_i)
method_i = body.find('method: "PATCH"', fetch_i)
body_i = body.find("JSON.stringify({ ignore_paths: lines })", method_i)
assert -1 < guard_i < parse_i < trim_i < drop_i < inflight_i, (
"guard → split + trim + drop blank lines → in-flight"
body_i = body.find(
"JSON.stringify({ ignore_paths: lines, ...(token ? { token } : {}) })", method_i
)
assert -1 < guard_i < parse_i < trim_i < drop_i < token_i < inflight_i, (
"guard → split + trim + drop blank lines → read the token → in-flight"
)
assert -1 < dis_c < dis_s < label_i < fetch_i < method_i < body_i, (
"disable both + 'Saving…' → the PATCH with the lines"
"disable both + 'Saving…' → the PATCH (lines + token only when non-blank)"
)
# Success: close → reload → announce (the exact order) — the
# update confirmation is the LAST announcement: the reload's