diff --git a/.agents/phases/todo/145_git_source_links/00_phase.md b/.agents/phases/complete/145_git_source_links/00_phase.md similarity index 100% rename from .agents/phases/todo/145_git_source_links/00_phase.md rename to .agents/phases/complete/145_git_source_links/00_phase.md diff --git a/.agents/phases/todo/145_git_source_links/01_origin_resolution.md b/.agents/phases/complete/145_git_source_links/01_origin_resolution.md similarity index 100% rename from .agents/phases/todo/145_git_source_links/01_origin_resolution.md rename to .agents/phases/complete/145_git_source_links/01_origin_resolution.md diff --git a/.agents/phases/todo/145_git_source_links/02_prompt_teaching.md b/.agents/phases/complete/145_git_source_links/02_prompt_teaching.md similarity index 100% rename from .agents/phases/todo/145_git_source_links/02_prompt_teaching.md rename to .agents/phases/complete/145_git_source_links/02_prompt_teaching.md diff --git a/.agents/phases/todo/145_git_source_links/03_content_payload_render_link.md b/.agents/phases/complete/145_git_source_links/03_content_payload_render_link.md similarity index 100% rename from .agents/phases/todo/145_git_source_links/03_content_payload_render_link.md rename to .agents/phases/complete/145_git_source_links/03_content_payload_render_link.md diff --git a/.agents/phases/todo/145_git_source_links/04_e2e_source_link.md b/.agents/phases/complete/145_git_source_links/04_e2e_source_link.md similarity index 100% rename from .agents/phases/todo/145_git_source_links/04_e2e_source_link.md rename to .agents/phases/complete/145_git_source_links/04_e2e_source_link.md diff --git a/.agents/reports/145_git_source_links/145_git_source_links__00_phase.a1.err b/.agents/reports/145_git_source_links/145_git_source_links__00_phase.a1.err new file mode 100644 index 0000000..e69de29 diff --git a/.agents/reports/145_git_source_links/145_git_source_links__00_phase.a1.md b/.agents/reports/145_git_source_links/145_git_source_links__00_phase.a1.md new file mode 100644 index 0000000..bcc195b --- /dev/null +++ b/.agents/reports/145_git_source_links/145_git_source_links__00_phase.a1.md @@ -0,0 +1,9 @@ +**Phase 145 final verification pass — all green, no code changes needed** + +- **Verified (no defects found):** resolver `origin_for_source` + `_url_basename` (scp/`.git`/trailing-slash, sanitize, ambiguity WARNING); `origin:`/`branch:` lines in all four read shapes via shared `_origin_lines`; HIGH `` origin rule (verbatim + only-byte-change pins); `DocContent.origin_url/origin_branch` (absent-never-null serializer); `renderDocument` "Original" anchor (copyable verbatim URL, `_blank`/`noopener`, branch in title); mock scripted origin-link flow (disjoint trigger — grep-verified vs all other E2E files) +- **Tests:** `uv run pytest --cov=app --cov-report=term-missing` → 3402 passed, `app/` coverage **99%** (>90% gate); `uv run pytest tests/e2e/test_git_source_links.py -v --no-cov` → **4 passed** in isolation; `uv run ruff check .` → clean; `uv run pyright` → 0 errors, 0 warnings +- **Completion criteria:** (1) git read results carry origin/branch, local byte-identical — ✅ (unit pins + E2E control); (2) HIGH `` rule pinned, LOW byte-identical — ✅; (3) modal AND /document.html show Original link, branch in label when pinned, nothing for local — ✅; (4) full suite green + coverage + E2E + lint/types — ✅; (5) no completed-phase behavior change (read header line 1, mock prefix, done frame/SourceRef untouched, LOW sha) — ✅ +- **Extra regression check:** re-ran shared-mock E2E suites `test_read_line_ranges.py` (2 passed) and `test_document_viewer.py` (8 passed) — no cross-suite regressions from the `mock_llm.py` addition +- **Documentation:** README.md reviewed in full (env table ⊇ `.env.example`, pages/routes, config defaults, scripts, 17-color palette, 300 s guard, 2 s sync poll, phase-145 provenance section matches implementation) and AGENTS.md (72rem container, no-CDN test, smoke test, gitignore, gates) — **verified consistent, no changes needed** +- **Notable:** phase-116 screenshot PNG was regenerated during this phase's E2E runs (byte re-encode only, visually identical, correct no-anchor local-source meta row) — left in tree for the harness commit +- **Next pending phase:** `146_search_tool_scope` (then 147, 148, 149) diff --git a/.agents/reports/145_git_source_links/145_git_source_links__00_phase.a1.validate b/.agents/reports/145_git_source_links/145_git_source_links__00_phase.a1.validate new file mode 100644 index 0000000..58d878c --- /dev/null +++ b/.agents/reports/145_git_source_links/145_git_source_links__00_phase.a1.validate @@ -0,0 +1,119 @@ +........................................................................ [ 2%] +........................................................................ [ 4%] +........................................................................ [ 6%] +........................................................................ [ 8%] +........................................................................ [ 10%] +........................................................................ [ 12%] +........................................................................ [ 14%] +........................................................................ [ 16%] +........................................................................ [ 19%] +........................................................................ [ 21%] +........................................................................ [ 23%] +........................................................................ [ 25%] +........................................................................ [ 27%] +........................................................................ [ 29%] +........................................................................ [ 31%] +........................................................................ [ 33%] +........................................................................ [ 36%] +........................................................................ [ 38%] +........................................................................ [ 40%] +........................................................................ [ 42%] +........................................................................ [ 44%] +........................................................................ [ 46%] +........................................................................ [ 48%] +........................................................................ [ 50%] +........................................................................ [ 52%] +........................................................................ [ 55%] +........................................................................ [ 57%] +........................................................................ [ 59%] +........................................................................ [ 61%] +........................................................................ [ 63%] +........................................................................ [ 65%] +........................................................................ [ 67%] +........................................................................ [ 69%] +........................................................................ [ 72%] +........................................................................ [ 74%] +........................................................................ [ 76%] +........................................................................ [ 78%] +........................................................................ [ 80%] +........................................................................ [ 82%] +........................................................................ [ 84%] +........................................................................ [ 86%] +........................................................................ [ 88%] +........................................................................ [ 91%] +........................................................................ [ 93%] +........................................................................ [ 95%] +........................................................................ [ 97%] +........................................................................ [ 99%] +............... [100%] +=============================== warnings summary =============================== +.venv/lib/python3.13/site-packages/fastapi/testclient.py:1 + /var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead. + from starlette.testclient import TestClient as TestClient # noqa + +-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html +================================ tests coverage ================================ +_______________ coverage: platform linux, python 3.13.13-final-0 _______________ + +Name Stmts Miss Cover +-------------------------------------------------- +app/__init__.py 1 0 100% +app/api/__init__.py 0 0 100% +app/api/auth.py 52 0 100% +app/api/chat.py 268 1 99% +app/api/chat_images.py 50 0 100% +app/api/chats.py 110 0 100% +app/api/config.py 13 0 100% +app/api/doc_drafts.py 99 0 100% +app/api/docs.py 181 1 99% +app/api/git_sources.py 259 0 100% +app/api/health.py 10 0 100% +app/api/steering.py 42 0 100% +app/api/suggestions.py 33 0 100% +app/api/sync.py 183 0 100% +app/api/tokens.py 40 0 100% +app/api/ui_settings.py 55 0 100% +app/config.py 264 0 100% +app/core/__init__.py 0 0 100% +app/core/auth.py 45 0 100% +app/core/caching.py 124 0 100% +app/core/debugging.py 29 2 93% +app/core/docs_push.py 39 0 100% +app/core/errors.py 5 0 100% +app/core/logging.py 13 0 100% +app/core/rate_limit.py 44 0 100% +app/core/security_headers.py 20 0 100% +app/core/theming.py 38 0 100% +app/core/tokens.py 44 0 100% +app/db.py 22 0 100% +app/main.py 68 0 100% +app/models.py 133 0 100% +app/rag/__init__.py 0 0 100% +app/rag/agent.py 431 1 99% +app/rag/archive_upload.py 134 0 100% +app/rag/chunker.py 220 4 98% +app/rag/doc_dates.py 18 0 100% +app/rag/folder_summaries.py 134 0 100% +app/rag/git_sources.py 77 0 100% +app/rag/importer.py 705 9 99% +app/rag/llm.py 246 1 99% +app/rag/md_images.py 63 0 100% +app/rag/overview.py 71 0 100% +app/rag/pdf.py 124 0 100% +app/rag/prompts.py 155 0 100% +app/rag/retriever.py 248 3 99% +app/rag/scaffolding.py 55 0 100% +app/rag/source_removal.py 41 0 100% +app/rag/sources_meta.py 16 0 100% +app/rag/suggestions.py 27 0 100% +app/rag/summarizer.py 58 0 100% +app/schemas.py 430 0 100% +-------------------------------------------------- +TOTAL 5537 22 99% +coverage gate: app/ 99% (>90%) OK +All checks passed! +0 errors, 0 warnings, 0 informations +WARNING: there is a new pyright version available (v1.1.411 -> v1.1.414). +Please install the new version or set PYRIGHT_PYTHON_FORCE_VERSION to `latest` + +validation OK diff --git a/.agents/reports/145_git_source_links/145_git_source_links__01_origin_resolution.a1.err b/.agents/reports/145_git_source_links/145_git_source_links__01_origin_resolution.a1.err new file mode 100644 index 0000000..e69de29 diff --git a/.agents/reports/145_git_source_links/145_git_source_links__01_origin_resolution.a1.md b/.agents/reports/145_git_source_links/145_git_source_links__01_origin_resolution.a1.md new file mode 100644 index 0000000..af425b7 --- /dev/null +++ b/.agents/reports/145_git_source_links/145_git_source_links__01_origin_resolution.a1.md @@ -0,0 +1,17 @@ +Task complete — all green. + +**Implemented** +- `app/rag/git_sources.py`: `_url_basename` (path component after `://` or scp-style `:`, trailing `/` + single `.git` stripped, last segment) and `origin_for_source(db, source_name)` beside `sanitize_url` — `kind='git'` rows only, exact-match → `(sanitize_url(url), branch)`, zero/two-or-more → `None` + WARNING (sanitized candidates), pure-DB (no git subprocess). +- `app/rag/agent.py`: shared `_origin_lines` helper; origin computed ONCE per successful read before the ranged/truncated branching; all four result builders insert `origin:`/`branch:` immediately after `date:` (before the image marker / `lines:` header); first line and all non-git results byte-identical. +- Tests: 4-shape × 4-origin read-result matrix in `tests/unit/test_agent.py` (exact strings, first-line `_READ_RESULT_PREFIX` contract, phase-122 marker pin, caplog ambiguity WARNING, real resolver driven via a git-only mock session) + resolver units in `tests/unit/test_git_sources.py` (basename mapping incl. scp-style/path-`@`, legacy `user:tok@` masked, zero/ambiguous `None`+WARNING, branch passthrough, local rows never origins). + +**Results** +- `uv run pytest --cov=app --cov-report=term-missing`: **3382 passed**, `app/` coverage **99%** (`git_sources.py` 100%, `agent.py` 99%) +- `uv run ruff check .` clean; `uv run pyright`: 0 errors +- Bonus: `uv run pytest tests/e2e/test_smoke.py -v --no-cov`: 3 passed (mock parse + SSE unaffected) + +**Docs**: no doc impact — model-internal read-result lines only; user-visible "Original" link arrives in task 03. + +**Decisions**: zero-match case also logs a WARNING (per task spec's "zero matches … → None + WARNING"); `origin:` sits above the `lines:` header in range shapes (strictly "immediately after the `date:` line"). + +**Next pending task**: `02_prompt_teaching.md` (the `` original-source rule + prompt pins). diff --git a/.agents/reports/145_git_source_links/145_git_source_links__01_origin_resolution.a1.validate b/.agents/reports/145_git_source_links/145_git_source_links__01_origin_resolution.a1.validate new file mode 100644 index 0000000..d3026d2 --- /dev/null +++ b/.agents/reports/145_git_source_links/145_git_source_links__01_origin_resolution.a1.validate @@ -0,0 +1,118 @@ +........................................................................ [ 2%] +........................................................................ [ 4%] +........................................................................ [ 6%] +........................................................................ [ 8%] +........................................................................ [ 10%] +........................................................................ [ 12%] +........................................................................ [ 14%] +........................................................................ [ 17%] +........................................................................ [ 19%] +........................................................................ [ 21%] +........................................................................ [ 23%] +........................................................................ [ 25%] +........................................................................ [ 27%] +........................................................................ [ 29%] +........................................................................ [ 31%] +........................................................................ [ 34%] +........................................................................ [ 36%] +........................................................................ [ 38%] +........................................................................ [ 40%] +........................................................................ [ 42%] +........................................................................ [ 44%] +........................................................................ [ 46%] +........................................................................ [ 48%] +........................................................................ [ 51%] +........................................................................ [ 53%] +........................................................................ [ 55%] +........................................................................ [ 57%] +........................................................................ [ 59%] +........................................................................ [ 61%] +........................................................................ [ 63%] +........................................................................ [ 65%] +........................................................................ [ 68%] +........................................................................ [ 70%] +........................................................................ [ 72%] +........................................................................ [ 74%] +........................................................................ [ 76%] +........................................................................ [ 78%] +........................................................................ [ 80%] +........................................................................ [ 83%] +........................................................................ [ 85%] +........................................................................ [ 87%] +........................................................................ [ 89%] +........................................................................ [ 91%] +........................................................................ [ 93%] +........................................................................ [ 95%] +........................................................................ [ 97%] +...................................................................... [100%] +=============================== warnings summary =============================== +.venv/lib/python3.13/site-packages/fastapi/testclient.py:1 + /var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead. + from starlette.testclient import TestClient as TestClient # noqa + +-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html +================================ tests coverage ================================ +_______________ coverage: platform linux, python 3.13.13-final-0 _______________ + +Name Stmts Miss Cover +-------------------------------------------------- +app/__init__.py 1 0 100% +app/api/__init__.py 0 0 100% +app/api/auth.py 52 0 100% +app/api/chat.py 268 1 99% +app/api/chat_images.py 50 0 100% +app/api/chats.py 110 0 100% +app/api/config.py 13 0 100% +app/api/doc_drafts.py 99 0 100% +app/api/docs.py 179 1 99% +app/api/git_sources.py 259 0 100% +app/api/health.py 10 0 100% +app/api/steering.py 42 0 100% +app/api/suggestions.py 33 0 100% +app/api/sync.py 183 0 100% +app/api/tokens.py 40 0 100% +app/api/ui_settings.py 55 0 100% +app/config.py 264 0 100% +app/core/__init__.py 0 0 100% +app/core/auth.py 45 0 100% +app/core/caching.py 124 0 100% +app/core/debugging.py 29 2 93% +app/core/docs_push.py 39 0 100% +app/core/errors.py 5 0 100% +app/core/logging.py 13 0 100% +app/core/rate_limit.py 44 0 100% +app/core/security_headers.py 20 0 100% +app/core/theming.py 38 0 100% +app/core/tokens.py 44 0 100% +app/db.py 22 0 100% +app/main.py 68 0 100% +app/models.py 133 0 100% +app/rag/__init__.py 0 0 100% +app/rag/agent.py 431 1 99% +app/rag/archive_upload.py 134 0 100% +app/rag/chunker.py 220 4 98% +app/rag/doc_dates.py 18 0 100% +app/rag/folder_summaries.py 134 0 100% +app/rag/git_sources.py 77 0 100% +app/rag/importer.py 705 9 99% +app/rag/llm.py 246 1 99% +app/rag/md_images.py 63 0 100% +app/rag/overview.py 71 0 100% +app/rag/pdf.py 124 0 100% +app/rag/prompts.py 154 0 100% +app/rag/retriever.py 248 3 99% +app/rag/scaffolding.py 55 0 100% +app/rag/source_removal.py 41 0 100% +app/rag/sources_meta.py 16 0 100% +app/rag/suggestions.py 27 0 100% +app/rag/summarizer.py 58 0 100% +app/schemas.py 424 0 100% +-------------------------------------------------- +TOTAL 5528 22 99% +coverage gate: app/ 99% (>90%) OK +All checks passed! +0 errors, 0 warnings, 0 informations +WARNING: there is a new pyright version available (v1.1.411 -> v1.1.414). +Please install the new version or set PYRIGHT_PYTHON_FORCE_VERSION to `latest` + +validation OK diff --git a/.agents/reports/145_git_source_links/145_git_source_links__02_prompt_teaching.a1.err b/.agents/reports/145_git_source_links/145_git_source_links__02_prompt_teaching.a1.err new file mode 100644 index 0000000..e69de29 diff --git a/.agents/reports/145_git_source_links/145_git_source_links__02_prompt_teaching.a1.md b/.agents/reports/145_git_source_links/145_git_source_links__02_prompt_teaching.a1.md new file mode 100644 index 0000000..69ac298 --- /dev/null +++ b/.agents/reports/145_git_source_links/145_git_source_links__02_prompt_teaching.a1.md @@ -0,0 +1,21 @@ +Task 02 complete. All gates 100% green. + +## Report + +**Implemented** +- `app/rag/prompts.py`: new canonical `ORIGIN_RULE` constant; inserted the locked verbatim original-source rule into `TOOLS_SECTION` at the `read` clause's end (after the truncation workflow sentence, before the `grep` clause). +- `app/rag/agent.py`: `AGENT_TOOLS` `read` description gains the one-clause pointer ("the result also carries an `origin` line (and `branch` when pinned) — include that original source in your answer (see your tools instructions)"). +- Pins: re-cut `TOOLS_SECTION` sha/len + kept pre-phase-145 anchors for a diff-style pin (rule is the *only* byte change); added verbatim+position, HIGH-only/LOW-absent, read-description-clause, and citation-discipline-unchanged tests. +- `README.md`: concise git-source-provenance note (origin/branch line + the answer link). + +**Results** +- `uv run pytest` → **3385 passed, 0 failed** +- `uv run pytest --cov=app` → **app/ 99%** (>90%); `prompts.py` 100%, `agent.py` 99% +- `uv run ruff check .` → all checks passed; `uv run pyright` → 0 errors/warnings +- `uv run pytest tests/e2e/test_smoke.py -v --no-cov` → 3 passed (mock regression check) + +**Docs**: README updated; AGENTS.md no impact (no command/config/convention change). + +**Decisions**: `ORIGIN_RULE` made the single source of truth, cross-pinned to the `TOOLS_SECTION` literal (removing it reproduces pre-phase bytes byte-for-byte); kept the LOW prompt and the mock's `` marker keying untouched; README describes only the prompt teaching + origin line (the UI "Original" anchor is task 03, so not over-claimed). + +**Next pending task**: `03_content_payload_render_link.md`. diff --git a/.agents/reports/145_git_source_links/145_git_source_links__02_prompt_teaching.a1.validate b/.agents/reports/145_git_source_links/145_git_source_links__02_prompt_teaching.a1.validate new file mode 100644 index 0000000..fdfdb73 --- /dev/null +++ b/.agents/reports/145_git_source_links/145_git_source_links__02_prompt_teaching.a1.validate @@ -0,0 +1,119 @@ +........................................................................ [ 2%] +........................................................................ [ 4%] +........................................................................ [ 6%] +........................................................................ [ 8%] +........................................................................ [ 10%] +........................................................................ [ 12%] +........................................................................ [ 14%] +........................................................................ [ 17%] +........................................................................ [ 19%] +........................................................................ [ 21%] +........................................................................ [ 23%] +........................................................................ [ 25%] +........................................................................ [ 27%] +........................................................................ [ 29%] +........................................................................ [ 31%] +........................................................................ [ 34%] +........................................................................ [ 36%] +........................................................................ [ 38%] +........................................................................ [ 40%] +........................................................................ [ 42%] +........................................................................ [ 44%] +........................................................................ [ 46%] +........................................................................ [ 48%] +........................................................................ [ 51%] +........................................................................ [ 53%] +........................................................................ [ 55%] +........................................................................ [ 57%] +........................................................................ [ 59%] +........................................................................ [ 61%] +........................................................................ [ 63%] +........................................................................ [ 65%] +........................................................................ [ 68%] +........................................................................ [ 70%] +........................................................................ [ 72%] +........................................................................ [ 74%] +........................................................................ [ 76%] +........................................................................ [ 78%] +........................................................................ [ 80%] +........................................................................ [ 82%] +........................................................................ [ 85%] +........................................................................ [ 87%] +........................................................................ [ 89%] +........................................................................ [ 91%] +........................................................................ [ 93%] +........................................................................ [ 95%] +........................................................................ [ 97%] +........................................................................ [ 99%] +. [100%] +=============================== warnings summary =============================== +.venv/lib/python3.13/site-packages/fastapi/testclient.py:1 + /var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead. + from starlette.testclient import TestClient as TestClient # noqa + +-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html +================================ tests coverage ================================ +_______________ coverage: platform linux, python 3.13.13-final-0 _______________ + +Name Stmts Miss Cover +-------------------------------------------------- +app/__init__.py 1 0 100% +app/api/__init__.py 0 0 100% +app/api/auth.py 52 0 100% +app/api/chat.py 268 1 99% +app/api/chat_images.py 50 0 100% +app/api/chats.py 110 0 100% +app/api/config.py 13 0 100% +app/api/doc_drafts.py 99 0 100% +app/api/docs.py 179 1 99% +app/api/git_sources.py 259 0 100% +app/api/health.py 10 0 100% +app/api/steering.py 42 0 100% +app/api/suggestions.py 33 0 100% +app/api/sync.py 183 0 100% +app/api/tokens.py 40 0 100% +app/api/ui_settings.py 55 0 100% +app/config.py 264 0 100% +app/core/__init__.py 0 0 100% +app/core/auth.py 45 0 100% +app/core/caching.py 124 0 100% +app/core/debugging.py 29 2 93% +app/core/docs_push.py 39 0 100% +app/core/errors.py 5 0 100% +app/core/logging.py 13 0 100% +app/core/rate_limit.py 44 0 100% +app/core/security_headers.py 20 0 100% +app/core/theming.py 38 0 100% +app/core/tokens.py 44 0 100% +app/db.py 22 0 100% +app/main.py 68 0 100% +app/models.py 133 0 100% +app/rag/__init__.py 0 0 100% +app/rag/agent.py 431 1 99% +app/rag/archive_upload.py 134 0 100% +app/rag/chunker.py 220 4 98% +app/rag/doc_dates.py 18 0 100% +app/rag/folder_summaries.py 134 0 100% +app/rag/git_sources.py 77 0 100% +app/rag/importer.py 705 9 99% +app/rag/llm.py 246 1 99% +app/rag/md_images.py 63 0 100% +app/rag/overview.py 71 0 100% +app/rag/pdf.py 124 0 100% +app/rag/prompts.py 155 0 100% +app/rag/retriever.py 248 3 99% +app/rag/scaffolding.py 55 0 100% +app/rag/source_removal.py 41 0 100% +app/rag/sources_meta.py 16 0 100% +app/rag/suggestions.py 27 0 100% +app/rag/summarizer.py 58 0 100% +app/schemas.py 424 0 100% +-------------------------------------------------- +TOTAL 5529 22 99% +coverage gate: app/ 99% (>90%) OK +All checks passed! +0 errors, 0 warnings, 0 informations +WARNING: there is a new pyright version available (v1.1.411 -> v1.1.414). +Please install the new version or set PYRIGHT_PYTHON_FORCE_VERSION to `latest` + +validation OK diff --git a/.agents/reports/145_git_source_links/145_git_source_links__03_content_payload_render_link.a1.err b/.agents/reports/145_git_source_links/145_git_source_links__03_content_payload_render_link.a1.err new file mode 100644 index 0000000..e69de29 diff --git a/.agents/reports/145_git_source_links/145_git_source_links__03_content_payload_render_link.a1.md b/.agents/reports/145_git_source_links/145_git_source_links__03_content_payload_render_link.a1.md new file mode 100644 index 0000000..2f160e8 --- /dev/null +++ b/.agents/reports/145_git_source_links/145_git_source_links__03_content_payload_render_link.a1.md @@ -0,0 +1,19 @@ +**Task 03 complete — report** + +**Implemented** +- `app/schemas.py`: `DocContent` gains `origin_url`/`origin_branch` (optional metadata, `image_url` style); serializer drops both when `None` — never `null` (non-git payload byte-identical) +- `app/api/docs.py`: `get_document_content` calls task-01's `origin_for_source(db, doc.source)` (same resolver as the read branch) and sets the two fields +- `frontend/assets/document.js`: new `originAnchor()` helper (`createElement("a")`, full URL verbatim as href+text, `target="_blank" rel="noopener"`, title = URL + ` — branch: ` when pinned); `renderDocument` inserts it after the path `code`, before the Created badge — one shared site for modal AND /document.html; absent → pre-phase meta row +- `frontend/assets/styles.css`: `.doc-origin` (`.doc-path` ellipsis idiom + `.doc-link` brand-ink affordance) + ≤640px 16rem squeeze +- Tests: 10 unit (model omission rule, endpoint stubbed git/pinned/sanitized/local/ambiguous, anchor source pins + node behavioral branch/no-branch variants, position pin, XSS contract, CSS pin) + 4 integration (seeded git row, pinned row, legacy credential sanitized end-to-end, plain local source → keys ABSENT) + +**Results** +- `uv run pytest --cov=app --cov-report=term-missing`: **3399 passed**, `app/` coverage **99%** (>90%) +- `uv run ruff check .`: clean; `uv run pyright`: **0 errors** +- E2E spot-runs in isolation: `tests/e2e/test_document_viewer.py` 8 passed, `tests/e2e/test_git_source_dates.py` 4 passed (screenshot visually identical — no anchor on the local fixture) + +**Docs**: README phase-145 section extended with the viewer's "Original" link; AGENTS.md — no doc impact (no command/config/convention change) + +**Notable decisions**: meta-row children built with a conditional spread (`...originItems`) so all pre-phase badge pins (sources_dates/date_editor/image_documents) pass untouched; existing E2E meta pins are selector-based and unaffected. Dev DB confirmed an empty test sandbox (integration truncation is the established pattern). + +**Next pending task**: `.agents/phases/todo/145_git_source_links/04_e2e_source_link.md` diff --git a/.agents/reports/145_git_source_links/145_git_source_links__03_content_payload_render_link.a1.validate b/.agents/reports/145_git_source_links/145_git_source_links__03_content_payload_render_link.a1.validate new file mode 100644 index 0000000..58d878c --- /dev/null +++ b/.agents/reports/145_git_source_links/145_git_source_links__03_content_payload_render_link.a1.validate @@ -0,0 +1,119 @@ +........................................................................ [ 2%] +........................................................................ [ 4%] +........................................................................ [ 6%] +........................................................................ [ 8%] +........................................................................ [ 10%] +........................................................................ [ 12%] +........................................................................ [ 14%] +........................................................................ [ 16%] +........................................................................ [ 19%] +........................................................................ [ 21%] +........................................................................ [ 23%] +........................................................................ [ 25%] +........................................................................ [ 27%] +........................................................................ [ 29%] +........................................................................ [ 31%] +........................................................................ [ 33%] +........................................................................ [ 36%] +........................................................................ [ 38%] +........................................................................ [ 40%] +........................................................................ [ 42%] +........................................................................ [ 44%] +........................................................................ [ 46%] +........................................................................ [ 48%] +........................................................................ [ 50%] +........................................................................ [ 52%] +........................................................................ [ 55%] +........................................................................ [ 57%] +........................................................................ [ 59%] +........................................................................ [ 61%] +........................................................................ [ 63%] +........................................................................ [ 65%] +........................................................................ [ 67%] +........................................................................ [ 69%] +........................................................................ [ 72%] +........................................................................ [ 74%] +........................................................................ [ 76%] +........................................................................ [ 78%] +........................................................................ [ 80%] +........................................................................ [ 82%] +........................................................................ [ 84%] +........................................................................ [ 86%] +........................................................................ [ 88%] +........................................................................ [ 91%] +........................................................................ [ 93%] +........................................................................ [ 95%] +........................................................................ [ 97%] +........................................................................ [ 99%] +............... [100%] +=============================== warnings summary =============================== +.venv/lib/python3.13/site-packages/fastapi/testclient.py:1 + /var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead. + from starlette.testclient import TestClient as TestClient # noqa + +-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html +================================ tests coverage ================================ +_______________ coverage: platform linux, python 3.13.13-final-0 _______________ + +Name Stmts Miss Cover +-------------------------------------------------- +app/__init__.py 1 0 100% +app/api/__init__.py 0 0 100% +app/api/auth.py 52 0 100% +app/api/chat.py 268 1 99% +app/api/chat_images.py 50 0 100% +app/api/chats.py 110 0 100% +app/api/config.py 13 0 100% +app/api/doc_drafts.py 99 0 100% +app/api/docs.py 181 1 99% +app/api/git_sources.py 259 0 100% +app/api/health.py 10 0 100% +app/api/steering.py 42 0 100% +app/api/suggestions.py 33 0 100% +app/api/sync.py 183 0 100% +app/api/tokens.py 40 0 100% +app/api/ui_settings.py 55 0 100% +app/config.py 264 0 100% +app/core/__init__.py 0 0 100% +app/core/auth.py 45 0 100% +app/core/caching.py 124 0 100% +app/core/debugging.py 29 2 93% +app/core/docs_push.py 39 0 100% +app/core/errors.py 5 0 100% +app/core/logging.py 13 0 100% +app/core/rate_limit.py 44 0 100% +app/core/security_headers.py 20 0 100% +app/core/theming.py 38 0 100% +app/core/tokens.py 44 0 100% +app/db.py 22 0 100% +app/main.py 68 0 100% +app/models.py 133 0 100% +app/rag/__init__.py 0 0 100% +app/rag/agent.py 431 1 99% +app/rag/archive_upload.py 134 0 100% +app/rag/chunker.py 220 4 98% +app/rag/doc_dates.py 18 0 100% +app/rag/folder_summaries.py 134 0 100% +app/rag/git_sources.py 77 0 100% +app/rag/importer.py 705 9 99% +app/rag/llm.py 246 1 99% +app/rag/md_images.py 63 0 100% +app/rag/overview.py 71 0 100% +app/rag/pdf.py 124 0 100% +app/rag/prompts.py 155 0 100% +app/rag/retriever.py 248 3 99% +app/rag/scaffolding.py 55 0 100% +app/rag/source_removal.py 41 0 100% +app/rag/sources_meta.py 16 0 100% +app/rag/suggestions.py 27 0 100% +app/rag/summarizer.py 58 0 100% +app/schemas.py 430 0 100% +-------------------------------------------------- +TOTAL 5537 22 99% +coverage gate: app/ 99% (>90%) OK +All checks passed! +0 errors, 0 warnings, 0 informations +WARNING: there is a new pyright version available (v1.1.411 -> v1.1.414). +Please install the new version or set PYRIGHT_PYTHON_FORCE_VERSION to `latest` + +validation OK diff --git a/.agents/reports/145_git_source_links/145_git_source_links__04_e2e_source_link.a1.err b/.agents/reports/145_git_source_links/145_git_source_links__04_e2e_source_link.a1.err new file mode 100644 index 0000000..e69de29 diff --git a/.agents/reports/145_git_source_links/145_git_source_links__04_e2e_source_link.a1.md b/.agents/reports/145_git_source_links/145_git_source_links__04_e2e_source_link.a1.md new file mode 100644 index 0000000..691a9d0 --- /dev/null +++ b/.agents/reports/145_git_source_links/145_git_source_links__04_e2e_source_link.a1.md @@ -0,0 +1,17 @@ +Task 04 complete — all gates 100% green. + +**Implemented** +- `tests/e2e/mock_llm.py`: new scripted `ORIGIN_LINK_TRIGGER` flow ("original source link", house style — docstring entry + constants + `_origin_link_url` + `_origin_link_flow` + dispatch before the plain tools flow). Request 1: scripted `read` (question-carried call); request 2: deterministic final answer — repo-URL sentence parsed by the mock FROM the read result's `origin:` line (the server-side half — the mock cannot fabricate it) + the read result VERBATIM (the rendered lens on the `origin:`/`branch:` lines); no-origin sentence for local docs; `Premise broke:` echo for refusals +- `tests/e2e/test_git_source_links.py`: dedicated Playwright suite, mock-only, isolated — 2-source KB (`skills` local dir whose name matches the seeded git row's URL basename `https://example.com/reese/skills` + `control` byte-identity dir), importer-in-thread, git row seeded per-test in the suite's own DB session (never at import time, so no clone); 4 tests: origin line in tool output + repo URL in done answer / modal `.doc-origin` anchor (href, `_blank`, `noopener`, verbatim URL text, bare title) / pinned-branch `title` (`… — branch: main`, text stays bare) + `branch: main` in the result / local control with NO `origin:` line and NO anchor + +**Results** +- `uv run pytest tests/e2e/test_git_source_links.py -v --no-cov` → **4 passed** (in isolation) +- Regression, in isolation: `tests/e2e/test_read_line_ranges.py` → 2 passed; `tests/e2e/test_document_viewer.py` → 8 passed +- `uv run pytest --cov=app --cov-report=term-missing` → **3399 passed**, `app/` coverage **99%** (>90%) +- `uv run ruff check .` → clean; `uv run pyright` → **0 errors** + +**Docs**: no doc impact (test-only task; README's phase-145 section was already accurate from tasks 02/03) + +**Decisions**: the phase-117 compact tool surface renders the *call* (not result text), so the origin-line assertion rides the mock's verbatim echo in the answer — the house scripted-turn lens; mock URL sentinel is `""` (homogeneous `tuple[str, …]`, pyright) + +**Next pending task**: none — 04 is phase 145's last task (harness performs the phase's final pass). diff --git a/.agents/reports/145_git_source_links/145_git_source_links__04_e2e_source_link.a1.validate b/.agents/reports/145_git_source_links/145_git_source_links__04_e2e_source_link.a1.validate new file mode 100644 index 0000000..58d878c --- /dev/null +++ b/.agents/reports/145_git_source_links/145_git_source_links__04_e2e_source_link.a1.validate @@ -0,0 +1,119 @@ +........................................................................ [ 2%] +........................................................................ [ 4%] +........................................................................ [ 6%] +........................................................................ [ 8%] +........................................................................ [ 10%] +........................................................................ [ 12%] +........................................................................ [ 14%] +........................................................................ [ 16%] +........................................................................ [ 19%] +........................................................................ [ 21%] +........................................................................ [ 23%] +........................................................................ [ 25%] +........................................................................ [ 27%] +........................................................................ [ 29%] +........................................................................ [ 31%] +........................................................................ [ 33%] +........................................................................ [ 36%] +........................................................................ [ 38%] +........................................................................ [ 40%] +........................................................................ [ 42%] +........................................................................ [ 44%] +........................................................................ [ 46%] +........................................................................ [ 48%] +........................................................................ [ 50%] +........................................................................ [ 52%] +........................................................................ [ 55%] +........................................................................ [ 57%] +........................................................................ [ 59%] +........................................................................ [ 61%] +........................................................................ [ 63%] +........................................................................ [ 65%] +........................................................................ [ 67%] +........................................................................ [ 69%] +........................................................................ [ 72%] +........................................................................ [ 74%] +........................................................................ [ 76%] +........................................................................ [ 78%] +........................................................................ [ 80%] +........................................................................ [ 82%] +........................................................................ [ 84%] +........................................................................ [ 86%] +........................................................................ [ 88%] +........................................................................ [ 91%] +........................................................................ [ 93%] +........................................................................ [ 95%] +........................................................................ [ 97%] +........................................................................ [ 99%] +............... [100%] +=============================== warnings summary =============================== +.venv/lib/python3.13/site-packages/fastapi/testclient.py:1 + /var/home/ducoterra/Projects/Personal/brain_of_reese/.venv/lib/python3.13/site-packages/fastapi/testclient.py:1: StarletteDeprecationWarning: Using `httpx` with `starlette.testclient` is deprecated; install `httpx2` instead. + from starlette.testclient import TestClient as TestClient # noqa + +-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html +================================ tests coverage ================================ +_______________ coverage: platform linux, python 3.13.13-final-0 _______________ + +Name Stmts Miss Cover +-------------------------------------------------- +app/__init__.py 1 0 100% +app/api/__init__.py 0 0 100% +app/api/auth.py 52 0 100% +app/api/chat.py 268 1 99% +app/api/chat_images.py 50 0 100% +app/api/chats.py 110 0 100% +app/api/config.py 13 0 100% +app/api/doc_drafts.py 99 0 100% +app/api/docs.py 181 1 99% +app/api/git_sources.py 259 0 100% +app/api/health.py 10 0 100% +app/api/steering.py 42 0 100% +app/api/suggestions.py 33 0 100% +app/api/sync.py 183 0 100% +app/api/tokens.py 40 0 100% +app/api/ui_settings.py 55 0 100% +app/config.py 264 0 100% +app/core/__init__.py 0 0 100% +app/core/auth.py 45 0 100% +app/core/caching.py 124 0 100% +app/core/debugging.py 29 2 93% +app/core/docs_push.py 39 0 100% +app/core/errors.py 5 0 100% +app/core/logging.py 13 0 100% +app/core/rate_limit.py 44 0 100% +app/core/security_headers.py 20 0 100% +app/core/theming.py 38 0 100% +app/core/tokens.py 44 0 100% +app/db.py 22 0 100% +app/main.py 68 0 100% +app/models.py 133 0 100% +app/rag/__init__.py 0 0 100% +app/rag/agent.py 431 1 99% +app/rag/archive_upload.py 134 0 100% +app/rag/chunker.py 220 4 98% +app/rag/doc_dates.py 18 0 100% +app/rag/folder_summaries.py 134 0 100% +app/rag/git_sources.py 77 0 100% +app/rag/importer.py 705 9 99% +app/rag/llm.py 246 1 99% +app/rag/md_images.py 63 0 100% +app/rag/overview.py 71 0 100% +app/rag/pdf.py 124 0 100% +app/rag/prompts.py 155 0 100% +app/rag/retriever.py 248 3 99% +app/rag/scaffolding.py 55 0 100% +app/rag/source_removal.py 41 0 100% +app/rag/sources_meta.py 16 0 100% +app/rag/suggestions.py 27 0 100% +app/rag/summarizer.py 58 0 100% +app/schemas.py 430 0 100% +-------------------------------------------------- +TOTAL 5537 22 99% +coverage gate: app/ 99% (>90%) OK +All checks passed! +0 errors, 0 warnings, 0 informations +WARNING: there is a new pyright version available (v1.1.411 -> v1.1.414). +Please install the new version or set PYRIGHT_PYTHON_FORCE_VERSION to `latest` + +validation OK diff --git a/.agents/screenshots/doc_modal_scrollbar_1280.png b/.agents/screenshots/doc_modal_scrollbar_1280.png index 88b7d35..54fc60a 100644 Binary files a/.agents/screenshots/doc_modal_scrollbar_1280.png and b/.agents/screenshots/doc_modal_scrollbar_1280.png differ diff --git a/README.md b/README.md index fcde376..1455d50 100644 --- a/README.md +++ b/README.md @@ -374,6 +374,21 @@ server-side tools — on **grounded** (high-relevance) turns only: (See [Answer images](#answer-images-phase-124) for the `show_image` tool.) +**Git-source provenance (phase 145).** When a document comes from a git +source, its `read` result also carries an `origin:` line (the repository's +URL, credentials stripped) and, when the source pins one, a `branch:` line. +The prompt teaches the model to include that original source in its answer — +a link to the repository with the document's path named (and the pinned +branch when one is shown) — so you can open the complete original in one +step rather than asking again for the source. Local (non-git) sources carry +no `origin:` line, so nothing is added and no link is invented. The same +origin rides the document-content payload (`origin_url` + `origin_branch` +when pinned), and the document viewer — the chat's doc modal AND the full +`/document.html` page, one render site — shows it as an **Original** +external link in the document's meta row (the full repository URL as the +link text, opening in a new tab; the pinned branch rides the link's +title). Non-git documents show nothing new. + Each call is executed against Postgres only (no extra LLM round trip) and streamed as an SSE `tool` frame. In the chat, each call shows a transient **calling-tool status** alongside "thinking" (the send button keeps its diff --git a/app/api/docs.py b/app/api/docs.py index a32efe0..46c9517 100644 --- a/app/api/docs.py +++ b/app/api/docs.py @@ -69,6 +69,7 @@ from app.models import Chunk, Document, FolderSummary from app.rag.agent import list_source_names from app.rag.doc_dates import normalize_doc_date from app.rag.folder_summaries import MIN_DOCS_PER_FOLDER, folder_of +from app.rag.git_sources import origin_for_source from app.rag.importer import match_extension from app.rag.llm import EmbeddingError, LLMClient from app.rag.summarizer import IMAGE_FALLBACK_MIME, IMAGE_MIMES @@ -181,6 +182,16 @@ def get_document_content( documents and a signed-in user (admin or token holder) opens a cited document by direct URL. The ONLY anonymous content left is the shared chats. + + Phase 145 (task 03): the payload carries the document's git origin + when one resolves — ``origin_url`` (the sanitized repository URL) + + ``origin_branch`` (only when the row pins one), computed by the + SAME :func:`app.rag.git_sources.origin_for_source` resolver the + agent's read branch uses (one mapping for the whole app: the + source's directory basename → the ``kind='git'`` row whose URL + basename matches). ``None`` — a non-git source, no row, or an + ambiguous basename — keeps the payload byte-identical to pre-phase + (both fields are dropped by the serializer, never ``null``). """ row = db.execute( select(Document, func.count(Chunk.id).label("chunks")) @@ -191,6 +202,11 @@ def get_document_content( if row is None: raise HTTPException(status_code=404, detail="document not found") doc, chunks = row + # Phase 145 (task 03): the document's git origin — the SAME resolver + # the read branch uses (one mapping for the whole app). ``None`` + # (non-git source, no row, ambiguous basename) → both fields are + # dropped on serialization, never ``null``. + origin = origin_for_source(db, doc.source) # Phase 122 (task 04): the image affordance — ``is_image`` is ALWAYS # present on the wire (text docs: false — the one new key); for an # image doc, ``image_url`` is the bytes route's path (absent for @@ -210,6 +226,8 @@ def get_document_content( image_url=( f"/api/documents/{doc.id}/image" if doc.is_image and doc.image_path else None ), + origin_url=origin[0] if origin else None, + origin_branch=origin[1] if origin else None, ) diff --git a/app/rag/agent.py b/app/rag/agent.py index 58b74f4..eb00559 100644 --- a/app/rag/agent.py +++ b/app/rag/agent.py @@ -280,7 +280,7 @@ from sqlalchemy.orm import Session from app.config import Settings from app.models import Document, FolderSummary from app.rag.folder_summaries import folder_of -from app.rag.git_sources import effective_sources +from app.rag.git_sources import effective_sources, origin_for_source from app.rag.llm import ( LLMClient, LLMError, @@ -362,8 +362,11 @@ AGENT_TOOLS: list[dict[str, Any]] = [ "first line) and `limit` (number of lines) to read " "exactly that part. Do not re-read a document you have " "already read in full — its full text is already in " - "your prompt. Call one tool at a time — wait for this " - "result before your next call." + "your prompt. For a document from a git source, the " + "result also carries an `origin` line (and `branch` " + "when pinned) — include that original source in your " + "answer (see your tools instructions). Call one tool at " + "a time — wait for this result before your next call." ), "parameters": { "type": "object", @@ -1272,6 +1275,32 @@ def _resolve_path(db: Session, combined: str) -> tuple[Document | None, str, str return find_document(db, source, path), source, path +def _origin_lines(origin: tuple[str, str | None] | None) -> str: + """Phase 145 (task 01) — the read result's origin lines, shared by + the four result builders (the lines are identical in every shape). + + ``None`` (the document's source has no unambiguous ``kind='git'`` + row — a local source, zero matches, or an ambiguous basename) → + ``""``: the result stays byte-identical to pre-phase. Otherwise + one ``origin: {url}`` line (the sanitized repository URL — the + phase-121 mask, so a legacy embedded-credential row's secret never + reaches the prompt) and, when a branch is pinned (non-NULL), one + ``branch: {branch}`` line. The builders insert the block + IMMEDIATELY AFTER the ``date:`` line and BEFORE the image marker + (the marker keeps its phase-122 position relative to the content, + the origin lines ride above it); the FIRST line + (``Document {source}/{path}:``) is untouched — the E2E mock's + ``_READ_RESULT_PREFIX`` header contract. + """ + if origin is None: + return "" + url, branch = origin + lines = f"origin: {url}\n" + if branch is not None: + lines += f"branch: {branch}\n" + return lines + + def all_documents(db: Session) -> list[Document]: """Every indexed document (full rows), ordered by ``(source, path)`` — catalog order. @@ -1728,6 +1757,14 @@ def _execute_tool( if (doc.source, doc.path) not in known: holder.read_docs.append(doc) holder.tool_calls += 1 + # Phase 145 (task 01): the git-sourced document's repository + # origin — computed ONCE per successful read, BEFORE the + # ranged/truncated branching (the lines are identical in all + # four result shapes, which each render the shared block via + # :func:`_origin_lines`). ``None`` (local source, no git row, + # ambiguous basename) → no lines: the result stays + # byte-identical to pre-phase. + origin = origin_for_source(db, doc.source) # Phase 122 (task 05): an image document's content IS the vision # description — the marker line (a third line between the # byte-identical header/date lines and the text) tells the model @@ -1775,6 +1812,7 @@ def _execute_tool( return ( f"Document {doc.source}/{doc.path}:\n" f"date: {doc.created_at:%Y-%m-%d}\n" + f"{_origin_lines(origin)}" f"{marker}" f"{doc.content[:cap]}\n" f"{TRUNCATION_MARKER}\n" @@ -1792,6 +1830,7 @@ def _execute_tool( return ( f"Document {doc.source}/{doc.path}:\n" f"date: {doc.created_at:%Y-%m-%d}\n" + f"{_origin_lines(origin)}" f"{marker}" f"{doc.content}" ) @@ -1829,6 +1868,7 @@ def _execute_tool( return ( f"Document {doc.source}/{doc.path}:\n" f"date: {doc.created_at:%Y-%m-%d}\n" + f"{_origin_lines(origin)}" f"lines: {first}-{end} of {total_lines}\n" f"{marker}{slice_text[:cap]}\n" f"{TRUNCATION_MARKER}\n" @@ -1842,6 +1882,7 @@ def _execute_tool( return ( f"Document {doc.source}/{doc.path}:\n" f"date: {doc.created_at:%Y-%m-%d}\n" + f"{_origin_lines(origin)}" f"lines: {first}-{end} of {total_lines}\n" f"{marker}{slice_text}" ) diff --git a/app/rag/git_sources.py b/app/rag/git_sources.py index db4229e..4668078 100644 --- a/app/rag/git_sources.py +++ b/app/rag/git_sources.py @@ -86,6 +86,83 @@ def sanitize_url(url: str) -> str: return _USERINFO_RE.sub(r"\1", url, count=1) +def _url_basename(url: str) -> str: + """Phase 145 (task 01) — the repository directory a git row's URL + imports into. + + The URL's path component — after the ``://``, or after the ``:`` + for scp-style ``git@host:owner/repo`` (the port-carrying + ``ssh://user@host:22/…`` form already fell into the ``://`` + branch) — with trailing ``/`` stripped, a single trailing ``.git`` + suffix stripped, and the last segment taken. The importer clones + into (and names the source after) exactly this basename + (``source = root.name``), so it is the join key between a + document's ``source`` name and the ``git_sources`` row whose clone + produced it. An ``@`` inside the *path* is not userinfo and is + never treated as one (no split, no strip). + """ + rest = url + if "://" in rest: + rest = rest.partition("://")[2] + elif ":" in rest: + # scp-style ``git@host:owner/repo`` — the path component starts + # after the colon (``host:port`` has no other form here: the + # only colon-carrying scheme without ``//`` is the scp one). + rest = rest.partition(":")[2] + rest = rest.rstrip("/") + if rest.endswith(".git"): + rest = rest[: -len(".git")] + return rest.rsplit("/", 1)[-1] + + +def origin_for_source(db: Session, source_name: str) -> tuple[str, str | None] | None: + """Phase 145 (task 01) — the origin a read result of *source_name* + documents carries: the sanitized repository URL (and the pinned + branch, when one exists), or ``None``. + + The candidate rows are the ``kind='git'`` rows (local rows are + never origins — nothing to link to from the user's browser); a row + matches when its URL basename (:func:`_url_basename` — the clone + directory the importer names the source after) equals + ``source_name``. + + * EXACTLY ONE match → ``(sanitize_url(row.url), row.branch)`` — + the phase-121 output mask keeps a legacy embedded-credential + row's secret out of the prompt (the stored value is never + modified — the :func:`clone_url_for` lesson), and ``branch`` is + the pinned branch or ``None`` (NULL = the default branch); + * zero matches, or TWO-OR-MORE (two repos sharing a basename) → + ``None`` + a WARNING log (the ambiguity names the sanitized + candidates) — never guess: the read result would otherwise + carry a wrong origin into every answer about the document, and + a ``None`` result is byte-identical to pre-phase. + + Pure-DB helper — one query, no git subprocess + (``scripts/git_sync.py`` stays the only git-invocation site, PLAN + §13), no caching (the table is a handful of rows; a read's extra + query is trivial). + """ + rows = db.scalars(select(GitSource).where(GitSource.kind == "git")).all() + candidates = [row for row in rows if _url_basename(row.url) == source_name] + if len(candidates) == 1: + row = candidates[0] + return sanitize_url(row.url), row.branch + if not candidates: + logger.warning( + "no git origin for source %r: no kind='git' row's URL basename matches", + source_name, + ) + return None + logger.warning( + "ambiguous git origin for source %r: %d rows share the URL basename (%s) — " + "no origin line is added to its read results", + source_name, + len(candidates), + ", ".join(sanitize_url(row.url) for row in candidates), + ) + return None + + def clone_url_for(row: GitSource) -> str: """Phase 121, LOCKED A2 — the URL git actually clones, with the row's credential injected ONLY here. diff --git a/app/rag/prompts.py b/app/rag/prompts.py index ad37581..06ef067 100644 --- a/app/rag/prompts.py +++ b/app/rag/prompts.py @@ -231,6 +231,40 @@ SUGGEST_INTRO = ( #: summaries are the starting points, not the content; ``grep`` with #: only ``pattern`` searches the whole knowledge base, and a source #: name is not a document. +#: Phase 145 (task 02) — the original-source rule the ``read`` teaching +#: carries (LOCKED verbatim — pinned in :mod:`tests.unit.test_prompt_lock` +#: and :mod:`tests.unit.test_prompts`). :data:`TOOLS_SECTION` embeds this +#: exact sentence-group at the ``read`` clause's end (the truncation +#: workflow sentence) and BEFORE the ``grep`` clause begins; the +#: cross-pins (``TOOLS_SECTION`` minus this rule reproduces the +#: pre-phase-145 bytes; ``count == 1`` + the boundary substring) keep +#: the embedded copy and this constant byte-identical. A read result for +#: a git-sourced document carries an ``origin:`` line (and a ``branch:`` +#: line when a branch is pinned) — the read contract's fact — so when +#: the answer uses such a document it includes the original source: a +#: link to the repository (the ``origin`` line) with the document's path +#: (the ``Document …`` header) named and the pinned branch when one is +#: shown, as the repository's file-view URL where the model can tell it +#: (``/blob//``), or the repository URL plus the +#: path as plain text otherwise — the user opens the complete original +#: from the answer in one step, not a second question. The rule is +#: CONDITIONAL on the ``origin`` line's presence — a local-sourced +#: document's read result carries none, so the rule adds nothing to +#: those answers (the model has no URL to cite; it must not invent one — +#: the sentence's conditioning does that job). +ORIGIN_RULE: str = ( + "A read result for a document from a git source carries an origin line " + "(and a branch line when a branch is pinned): when your answer uses " + "such a document, include the original source in the answer — a link " + "to the repository from the origin line, with the document's path " + "(from the Document header) named and the pinned branch when one is " + "shown — as the repository's file-view URL where you can tell it " + "(for example /blob//), or the repository " + "URL plus the path as plain text otherwise. The user should be able " + "to open the complete original from your answer in one step, not ask " + "again for the source." +) + TOOLS_SECTION: str = ( "\n" "You may extend your context with three tools. `ls` lists the " @@ -260,7 +294,17 @@ TOOLS_SECTION: str = ( "are capped: a " "cut read returns the first part plus a TRUNCATED notice — the " "document did not end where it stopped; use `grep` (pattern) to " - "find the rest, it searches the whole document. `grep` locates an exact string " + "find the rest, it searches the whole document. A read result for a " + "document from a git source carries an origin line (and a branch line " + "when a branch is pinned): when your answer uses such a document, " + "include the original source in the answer — a link to the " + "repository from the origin line, with the document's path (from the " + "Document header) named and the pinned branch when one is shown — as " + "the repository's file-view URL where you can tell it (for example " + "/blob//), or the repository URL plus the " + "path as plain text otherwise. The user should be able to open the " + "complete original from your answer in one step, not ask again for " + "the source. `grep` locates an exact string " "(case-insensitive) in the indexed documents and returns up to 20 " "matching `source/path:line: text` lines — a locator, not a " "context-adder: read the winner with `read`. A grep pattern is a " diff --git a/app/schemas.py b/app/schemas.py index 6a43e37..aa33943 100644 --- a/app/schemas.py +++ b/app/schemas.py @@ -561,12 +561,30 @@ class DocContent(BaseModel): #: omission precedent); also absent for an image row whose #: ``image_path`` is NULL (the viewer's onerror fallback covers it). image_url: str | None = None + #: Phase 145 (task 03) — the document's git origin: the sanitized + #: repository URL (the task-01 resolver's output — the phase-121 + #: credential-free mask; the document's ``source`` is the directory + #: the clone landed in, so no unambiguous git row means no origin). + #: ABSENT from the wire for non-git documents (``None`` → dropped by + #: the serializer — never ``null``, the :func:`_drop_absent_share_url` + #: omission precedent), so a non-git document's payload is + #: byte-identical to pre-phase. + origin_url: str | None = None + #: Phase 145 (task 03) — the row's PINNED branch (phase 135). + #: ABSENT from the wire when the row pins none (NULL = the + #: repository's default branch — nothing to show); also dropped by + #: the serializer (never ``null``). + origin_branch: str | None = None @model_serializer(mode="wrap") def _serialize(self, handler: SerializerFunctionWrapHandler) -> Any: data = handler(self) if self.image_url is None: data.pop("image_url", None) + if self.origin_url is None: + data.pop("origin_url", None) + if self.origin_branch is None: + data.pop("origin_branch", None) return data diff --git a/frontend/assets/document.js b/frontend/assets/document.js index 9ee5a1e..d9fb3e0 100644 --- a/frontend/assets/document.js +++ b/frontend/assets/document.js @@ -118,6 +118,22 @@ * the row exists but the copy was lost) swaps the block for a small * "Image unavailable" note (role=status): the page still shows the * description. ONE shared core — page + modal both get it. + * + * Phase 145 (task 03): the meta row's "Original" anchor — when the + * content payload carries origin_url (a git-sourced document — the + * task-01 resolver's sanitized repository URL; ABSENT on the wire + * for non-git documents, whose meta row stays byte-identical to + * pre-phase), an joins the .doc-meta row + * AFTER the path code and BEFORE the Created badge: href + visible + * text = the full URL VERBATIM (the copyable-link contract — no + * decorative label inside the anchor text; the .doc-origin ellipsis + * clips the display), target=_blank + rel=noopener (the modal's + * "Full page" escape-hatch idiom), and the title carries the full + * URL plus ` — branch: ` when the payload pins a branch (the + * pathCode ellipsis-precision idiom). ONE shared core — the modal + * AND /document.html render it (no per-surface copy); createElement + * + properties/textContent only (the never-build-HTML rule — no + * innerHTML). */ import { bindSharedHeaderControls, fetchIsAdmin, initSharedHeader } from "./header.js"; @@ -158,6 +174,32 @@ function metaBadge(cls, text, title) { return el; } +/* Phase 145 (task 03): the meta row's "Original" anchor — the git + * origin as a copyable external link (called only when the payload + * carries origin_url — a non-git document never gets one, so its + * meta row stays the pre-phase shape). The FULL URL is the anchor's + * visible text (the copyable-link contract — no decorative label + * inside the anchor text: a " (branch)" suffix would break + * copy-paste; the .doc-origin CSS ellipsis clips the display + * instead), target=_blank + rel=noopener (the modal's "Full page" + * escape-hatch idiom), and the title carries the full URL — plus + * ` — branch: ` when the payload pins a branch (the pathCode + * ellipsis-precision idiom). Properties + textContent only — never + * innerHTML (the XSS contract). */ +function originAnchor(doc) { + const a = document.createElement("a"); + a.className = "doc-origin"; + a.href = doc.origin_url; + a.target = "_blank"; + a.rel = "noopener"; + a.textContent = doc.origin_url; + a.setAttribute( + "title", + doc.origin_branch ? doc.origin_url + " — branch: " + doc.origin_branch : doc.origin_url, + ); + return a; +} + /* ---------- shared renderer (phase 26, task 02) ---------- * Populates the three elements every render surface provides: a title, * a .doc-meta badge row (source · format · mono path · created · @@ -178,10 +220,20 @@ export function renderDocument(doc, { titleEl, metaEl, contentEl }) { const pathCode = document.createElement("code"); pathCode.className = "doc-path"; pathCode.textContent = doc.path; + // Phase 145 (task 03): the "Original" anchor — the git origin as a + // copyable external link, present ONLY when the content payload + // carries origin_url (a git-sourced document's sanitized repository + // URL — the task-01 resolver's output; ABSENT on the wire for + // non-git documents, so their meta row stays byte-identical to + // pre-phase). It lands AFTER the path code and BEFORE the Created + // badge — one more inline item in the row (the modal's wrap rule + // and the page's clip rule both keep it on their shapes). + const originItems = doc.origin_url ? [originAnchor(doc)] : []; metaEl.replaceChildren( metaBadge("doc-source-badge", doc.source), metaBadge("format-badge", doc.format), pathCode, + ...originItems, // Phase 106 (task 08, D8): the Created badge BEFORE the Indexed // one (the owner's verbatim position — the date at the top of a // clicked document). textContent = the locale date (fmtDate — the diff --git a/frontend/assets/styles.css b/frontend/assets/styles.css index 00fa8e1..398bf77 100644 --- a/frontend/assets/styles.css +++ b/frontend/assets/styles.css @@ -4400,6 +4400,28 @@ a.answer-image:hover { text-decoration: underline; } /* flat — the caption und keeps a visible box on narrow screens while the trailing badges clip. */ min-width: 6rem; } +/* Phase 145 (task 03): the "Original" anchor — the git origin as a + copyable external link in the meta row (rendered by the ONE + renderDocument core — the modal AND /document.html; a non-git + document never gets one). The .doc-path code family (mono, 0.75rem, + the SAME 26rem ellipsis budget + 6rem floor — the row clips, the + title carries the full URL) with the .doc-link affordance (brand-ink + 8.7:1 on --surface — the recorded .doc-link pair; underline on + hover/focus, the .doc-link idiom). The anchor's visible text IS the + full URL (the copyable-link contract — the ellipsis clips the + display). */ +.doc-origin { + font-family: var(--mono); + font-size: 0.75rem; + max-width: 26rem; + white-space: nowrap; + overflow: hidden; + text-overflow: ellipsis; + min-width: 6rem; + color: var(--brand-ink); /* 8.7:1 on --surface (the .doc-link pair) */ + text-decoration: none; +} +.doc-origin:hover, .doc-origin:focus-visible { text-decoration: underline; } .doc-chunks { font-family: var(--mono); } /* Phase 106 (task 08, D8): the Created badge — the SAME family as the Indexed one (no own look: the meta row's text, var(--ink-soft) on @@ -5447,6 +5469,7 @@ a.answer-image:hover { text-decoration: underline; } /* flat — the caption und .doc-titlebar .container { gap: 0.5rem; } /* phase 34: row 2 squeezes like row 1 at mobile widths */ #doc-title { font-size: 1.1rem; } .doc-path { max-width: 16rem; } + .doc-origin { max-width: 16rem; } /* phase 145 (task 03): the Original anchor squeezes like the path */ .doc-md { padding: 1.1rem 1rem; } .doc-summary { padding: 0.75rem 0.9rem; } .doc-raw { padding: 1rem; font-size: 0.8rem; } diff --git a/tests/e2e/mock_llm.py b/tests/e2e/mock_llm.py index a68913f..92c6f9f 100644 --- a/tests/e2e/mock_llm.py +++ b/tests/e2e/mock_llm.py @@ -714,6 +714,55 @@ Implements just enough of the aipi surface: deflected turns never hit it); verified 2026-10-01: no existing E2E question or fixture file contains the phrase, so every other suite is unaffected. + - user message containing ``original source link`` + (``ORIGIN_LINK_TRIGGER``, phase 145 task 04 — the git-source- + links' dedicated story suite + ``tests/e2e/test_git_source_links.py``) **and** the system + prompt carries the ```` section -> the deterministic + SCRIPTED ORIGIN-LINK flow (the owner's scenario, TODO.md L7: + the answer grounded in a git-sourced document must carry the + way to the complete original in ONE step): the question + carries its own tool call after the colon — ``original source + link: read source/path`` — the question-carried-call + convention (``_READ_CAP_CALL_RE``), then discriminated + statelessly from the tool results (streaming only): + * request 1 (``tools`` offered, no tool results yet): a + ``tool_calls`` delta — ``read`` on the parsed target + (synthetic id ``call_0``); + * a ``tool``-role READ result is in the messages (content + starting with the agent's ``"Document "`` prefix — + ``_READ_RESULT_PREFIX``; the phase-145 ``origin:`` / + ``branch:`` lines sit BELOW the header, so the first-line + parsing is untouched): the deterministic FINAL answer — + the repository URL VERBATIM, parsed from the read + result's ``origin:`` line BY THE MOCK ITSELF (the mock + cannot fabricate it — the real server's resolver put the + line in the result, so the URL in the answer is the + server-side half's assertion): ``The full source lives at + — see the repo for the complete original.`` + (``ORIGIN_LINK_ANSWER``) followed by the house + scripted-turn echo — the read result VERBATIM under + ``Here's what the read returned:`` (the rendered output + is the E2E's lens on the ``origin:``/``branch:`` lines' + presence); + * a tool result is in the messages that carries NO + ``origin:`` line (a local-sourced document — the byte- + identity control): the deterministic NO-ORIGIN answer — + ``No origin line in the read result — the document is not + from a git source.`` (``ORIGIN_LINK_NO_ORIGIN``) + the + same verbatim echo (the suite asserts the line's ABSENCE + through it); + * a tool result is in the messages that is NOT a read + result (a refusal — the premise broke): the + deterministic broken-premise echo — the offending result + VERBATIM under ``Premise broke:`` (the two-ranges + convention — the suite fails loudly on it). + Checked BEFORE the plain ``TOOLS_TRIGGER`` flow (disjoint + trigger phrases — the phase-71/72/94/118/131/139/141 ordering + convention; the trigger needs the ```` section, so + deflected turns never hit it); verified 2026-10-01: no + existing E2E question or fixture file contains the phrase, so + every other suite is unaffected. Failure injection (phase 67, LLM retry, TODO.md L3) — deterministic dead-endpoint behavior for the retry E2E suite (``tests/e2e/ @@ -2667,6 +2716,148 @@ def _two_ranges_flow(body: dict[str, Any]) -> tuple[str, ...] | None: return ("read1", target) +# --------------------------------------------------------------------------- +# Phase 145 (task 04): the git-source-links' dedicated story suite — the +# owner's "the answer points at the original git source" scenario. See +# the module docstring. +# --------------------------------------------------------------------------- + +#: A user message containing this substring (case-insensitive) — +#: combined with the ```` section in the system prompt — +#: drives the deterministic SCRIPTED ORIGIN-LINK flow (the +#: git-source-links' story suite, +#: ``tests/e2e/test_git_source_links.py`` — the owner's scenario, +#: TODO.md L7: the answer grounded in a git-sourced document must +#: carry the way to the complete original in ONE step). Checked +#: BEFORE the plain ``TOOLS_TRIGGER`` flow (disjoint trigger +#: phrases — the phase-71/72/94/118/131/139/141 ordering +#: convention; the trigger needs the ```` section, so +#: deflected turns never hit it); verified 2026-10-01: no +#: existing E2E question or fixture file contains the phrase, so +#: every other suite is unaffected. +ORIGIN_LINK_TRIGGER = "original source link" + +#: The scripted call in the origin-link question (case-insensitive +#: — the suite's questions capitalize the trigger's first +#: letter): the verb (``read``) plus the target — a combined +#: ``source/path``, parsed from the RAW user message so the target +#: keeps its case. The target is a ``[a-z0-9_./-]`` run (case- +#: insensitively), so the suite's `` — `` flavor separator (em +#: dash) can never bleed into it (the read-cap / drill-down +#: convention, ``_READ_CAP_CALL_RE`` / ``_DRILL_CALL_RE``). +_ORIGIN_LINK_CALL_RE = re.compile( + r"original source link:\s*read\s+(?P[a-z0-9_./-]+)", + re.I, +) + +#: The flow's FINAL answer for a read result that carries the +#: phase-145 ``origin:`` line — the DETERMINISTIC string carrying +#: the repository URL VERBATIM (the model-side contract, owner- +#: confirmed 2026-09-30: the answer includes the original source). +#: ``{url}`` is parsed from the read result's ``origin:`` line BY +#: THE MOCK ITSELF — the mock cannot fabricate it, so the URL in +#: the answer is the server-side half's assertion (the real +#: server's resolver put the line in the result the model saw). +ORIGIN_LINK_ANSWER = ( + "The full source lives at {url} — see the repo for the complete original." +) + +#: The flow's FINAL answer for a read result that carries NO +#: ``origin:`` line (a local-sourced document — the byte-identity +#: control): the deterministic no-origin sentence (its wording +#: carries NO ``origin:`` substring — the suite's absence +#: assertion keys off the line's exact prefix). +ORIGIN_LINK_NO_ORIGIN = ( + "No origin line in the read result — the document is not from a git source." +) + +#: The house scripted-turn echo header (the ``READ_CAP_TRIGGER`` +#: flow's convention): the read result rides VERBATIM into the +#: answer, so the rendered output is the E2E's lens on the +#: ``origin:``/``branch:`` lines' presence (and their absence in +#: the local-source control). +_ORIGIN_LINK_ECHO_HEADER = "Here's what the read returned:" + + +def _origin_link_url(result: str) -> str | None: + """The read result's ``origin:`` line's URL (the phase-145 + task-01 line — the sanitized repository URL the real server's + resolver put in the result) — or ``None`` when the result + carries no origin line (a local-sourced document, no git row, + or an ambiguous basename). + + The search is position-independent (any line with the exact + ``origin: `` prefix): the phase-145 lines sit between the + ``date:`` line and the content, but the parse never drifts if + the result's header shape changes. A document BODY line that + merely starts with ``origin: `` would be a fixture defect (the + suite's fixtures are pinned content — no such line), never a + mock defect. + """ + for line in result.splitlines(): + if line.startswith("origin: "): + return line[len("origin: "):] + return None + + +def _origin_link_flow(body: dict[str, Any]) -> tuple[str, ...] | None: + """Classify a phase-145 scripted origin-link request (see the + module docstring). The question carries the scripted call + (``original source link: read source/path``); the step is then + discriminated statelessly from the tool results, like the + other marker flows: + + * ``("call", target, "call_0")`` — ``tools`` are offered and + no ``tool``-role result is in the messages yet: the scripted + ``read`` on the parsed target (synthetic id ``call_0``). + * ``("answer", url, result)`` — a ``tool``-role READ result + (content starting with the agent's ``"Document "`` prefix — + ``_READ_RESULT_PREFIX``; the phase-145 ``origin:`` / + ``branch:`` lines sit BELOW the header, so the first-line + parsing is untouched) is in the messages: the deterministic + FINAL answer — ``url`` the result's ``origin:`` line's URL + (or ``""`` — the no-origin sentence, the local-source + control; the empty-string sentinel keeps the return + homogeneous ``tuple[str, …]``, the ``_read_lines_flow`` + convention) and ``result`` the read result VERBATIM (the + house scripted-turn echo — the rendered output is the lens + on the origin lines). + * ``("bad", result)`` — a tool result is in the messages that + is NOT a read result (a refusal — the premise broke): the + deterministic broken-premise echo — the offending result + VERBATIM under ``Premise broke:`` (the two-ranges convention + — the suite fails loudly on it). + * ``None`` — not the flow: the trigger is absent, the + ```` section is missing (deflected turns never carry + it), the scripted call is unparseable, or ``tools`` are not + offered and no tool results are in the messages yet (e.g. + ``agent_max_rounds=0``). + """ + user = _user(body) + if ORIGIN_LINK_TRIGGER not in user.lower(): + return None + if "" not in _system(body): + return None + match = _ORIGIN_LINK_CALL_RE.search(user) + if match is None: + return None + target = match.group("arg") + results = _tool_results(body) + if results: + last = results[-1] + if not last.startswith(_READ_RESULT_PREFIX): + return ("bad", last) + # The empty-string sentinel (never a real URL — a URL is + # always non-empty: the resolver returns the stored URL's + # sanitized form, which is a non-empty string): keeps the + # return homogeneous ``tuple[str, …]`` (pyright), the + # ``_read_lines_flow`` convention. + return ("answer", _origin_link_url(last) or "", last) + if not body.get("tools"): + return None + return ("call", target, "call_0") + + def long_answer() -> str: """~900-word deterministic walkthrough (phase 11): numbered steps plus a unique final line that must survive the stream untruncated.""" @@ -4044,6 +4235,57 @@ def chat_completions(body: dict[str, Any]) -> Any: media_type="text/event-stream", headers={"Cache-Control": "no-cache", "X-Accel-Buffering": "no"}, ) + # Phase 145 (task 04): the owner's "the answer points at the + # original git source" scenario (the git-source-links' story + # suite, tests/e2e/test_git_source_links.py — TODO.md L7): + # the scripted read, then the final answer that carries the + # repository URL parsed from the read result's ``origin:`` + # line (the server-side half — the mock cannot fabricate it) + # + the result VERBATIM (the lens on the origin lines' + # presence/absence). Checked BEFORE the plain TOOLS_TRIGGER + # flow (disjoint trigger phrases — the phase-71/72/94/118/ + # 131/139/141 ordering convention; the trigger needs the + # ```` section, so deflected turns never hit it). + origin_link = _origin_link_flow(body) + if origin_link is not None: + if origin_link[0] == "call": + stream = _tool_call_stream( + "read", {"path": origin_link[1]}, origin_link[2] + ) + elif origin_link[0] == "bad": + # The premise broke (a refusal — the scripted read + # was refused): echo the offending result verbatim + # so a suite fails loudly on it (the two-ranges + # convention). + stream = _sse_stream( + _apply_max_tokens( + f"Premise broke:\n{origin_link[1]}", + body.get("max_tokens"), + ), + 0.0, + ) + else: # "answer" — the final answer (tests 1–3's model- + # side contract) + the verbatim echo (the lens on + # the origin lines' presence/absence). ``url`` is + # the ``origin:`` line's URL or the "" sentinel + # (the no-origin sentence — the local-source + # control). + url = origin_link[1] + sentence = ( + ORIGIN_LINK_ANSWER.format(url=url) if url else ORIGIN_LINK_NO_ORIGIN + ) + stream = _sse_stream( + _apply_max_tokens( + f"{sentence}\n{_ORIGIN_LINK_ECHO_HEADER}\n{origin_link[2]}", + body.get("max_tokens"), + ), + 0.0, + ) + return StreamingResponse( + stream, + media_type="text/event-stream", + headers={"Cache-Control": "no-cache", "X-Accel-Buffering": "no"}, + ) # Phase 124 (task 07, image chips and answers, TODO L3): the # scripted SHOW_IMAGE flow — the mock shows the seeded image # document (the ```` block with the image- diff --git a/tests/e2e/test_git_source_links.py b/tests/e2e/test_git_source_links.py new file mode 100644 index 0000000..09b32a3 --- /dev/null +++ b/tests/e2e/test_git_source_links.py @@ -0,0 +1,585 @@ +"""Phase 145 task 04 E2E (Playwright, mock-only): the git-source-links' +dedicated story suite — the owner's scenario, end to end. + +The browser-visible proof (TODO.md L7: "there always needs to be a +link to the original git source if possible … point to the skills +repo in the answer so I can go download the skills"): a question +about a git-sourced document produces (1) an answer that contains +the repository link (the model-side teaching, through the real read +result that carries the ``origin:`` line) and (2) a document modal +whose meta row carries the "Original" external anchor. A +local-sourced document produces neither (the byte-identity guard). + +Run in isolation (DB must be up: ``podman compose up -d db``): + + uv run pytest tests/e2e/test_git_source_links.py -v --no-cov + +MOCK-ONLY suite: ``E2E_REAL_LLM=1`` is not supported — the real +``turbo`` does whatever it does with the tools, while this story's +gate is the deterministic SCRIPTED flow in ``tests/e2e/mock_llm.py`` +(the mock is the only E2E lens on the LLM's context): + +* ``ORIGIN_LINK_TRIGGER`` — a user message containing ``original + source link`` **and** the system prompt carrying the HIGH + prompt's ```` section. The question carries its own + scripted call after the colon — ``original source link: read + source/path`` (the question-carried-call convention, + ``_READ_CAP_CALL_RE``). Request 1 is the scripted ``read``; + request 2 (the read result present) is the deterministic FINAL + answer — the repository URL VERBATIM, parsed from the read + result's ``origin:`` line BY THE MOCK (it cannot fabricate it — + the real server's resolver put the line in the result, so the + URL in the answer is the server-side half's assertion) — + followed by the house scripted-turn echo (the read result + VERBATIM — the rendered output is the lens on the + ``origin:``/``branch:`` lines' presence; the local-source + control's read result carries none, and the mock answers with + its deterministic no-origin sentence instead). + +KB fixture — TWO local directories (the house per-module import — +the ``test_document_viewer.py`` importer-in-a-thread pattern), each +holding ONE readable markdown document: + +* ``skills/phased-execution.md`` — the owner's scenario document + (the phased execution process for pi.dev). The source's name is + ``skills`` — it MATCHES the seeded git row's URL basename + (``https://example.com/reese/skills`` → ``skills``: the path + component, trailing ``/`` stripped, no ``.git`` suffix to strip, + last segment — the resolver's name mapping). The fixture document + is locally imported, but the mapping is name-based — which is the + point of the seam; +* ``control/notes.md`` — a SECOND plain local source with NO git + row: the byte-identity control (its read result and its modal + meta row must be exactly the pre-phase shapes). + +The git row is seeded in the suite's OWN DB session (a direct +``SessionLocal`` write, per test — ``branch=None`` for tests 1–2, +``branch="main"`` for the pinned-branch test, and ABSENT for the +control test) — never through the API, and never during the import +(a ``kind='git'`` row present at import time would be cloned — +``example.com`` is not a git host; the resolver reads the table +LIVE at read/content time, so seeding after the import is the +whole trick). + +The app under test is the conftest session app (mock LLM + the +mock-calibrated gate: threshold 0.30 + lexical floor 0.15): both +questions share the subject words with their own fixture document +(the mock's token-overlap distribution), so every scripted turn +stays GROUNDED (the ```` section is present) and the read +document is the turn's cited chip (phase 119 A1 — the done frame's +``sources`` is the READ docs). + +Test → observable mapping (Playwright Mapping Rule; the story is +the owner's TODO item — one Playwright file per story, A16): +1. ``test_answer_carries_origin_link`` — the ``original source + link`` question on the git-matched source's document → the + rendered tool output for the ``read`` (the mock's verbatim echo + in the answer — the house scripted-turn lens) carries + ``origin: https://example.com/reese/skills`` (the server put + the line in the result the model saw) and the final ``done`` + answer contains the repository URL (the scripted echo — the + model-side contract, deterministic). +2. ``test_modal_shows_original_anchor`` — same setup → click the + source chip → ``#doc-modal`` rendered → the meta row contains + the ``.doc-origin`` anchor with ``href`` = the URL, + ``target="_blank"``, ``rel="noopener"``, ``textContent`` = the + URL verbatim (and NO branch in the text — ``branch=None``: the + ``title`` is the bare URL). +3. ``test_pinned_branch_in_title`` — the ``branch="main"`` row → + the read result's surface also carries ``branch: main``, the + done answer still carries the repository URL, and the same + anchor's ``title`` is + ``https://example.com/reese/skills — branch: main`` (the text + stays the bare URL — the copyable-link contract). +4. ``test_local_source_has_no_origin`` — the control source's + document (no git rows at all — the pre-phase world) → the + read's rendered tool output carries NO ``origin:`` line (the + byte-identical result shape — the mock's no-origin sentence + answers) and the modal's meta row has NO ``.doc-origin`` + anchor (the pre-phase badge row). +""" +from __future__ import annotations + +import asyncio +import json +import time +from collections.abc import Iterator +from pathlib import Path +from threading import Thread +from typing import Any + +import pytest +from playwright.sync_api import Page, expect +from sqlalchemy import delete, func, select, text + +from app.config import Settings +from app.db import SessionLocal +from app.models import Document, GitSource +from app.rag.importer import import_sources +from app.rag.llm import LLMClient +from e2e.auth_helpers import login +from e2e.mock_llm import ORIGIN_LINK_ANSWER, ORIGIN_LINK_NO_ORIGIN + +REPO = Path(__file__).resolve().parents[2] + +SOURCE = "skills" # the local directory's basename = the source name +SKILLS_REL = "phased-execution.md" +SKILLS_SP = f"{SOURCE}/{SKILLS_REL}" # the combined read identity +CONTROL = "control" # the plain local control source (NO git row) +CONTROL_REL = "notes.md" +CONTROL_SP = f"{CONTROL}/{CONTROL_REL}" + +#: The seeded git row's public URL (no credential — the public form; +#: the phase-121 sanitizer is a byte-identical no-op for it). Its +#: URL basename — ``skills`` — MATCHES the fixture source's name +#: (the resolver's name mapping — the seam). +ORIGIN_URL = "https://example.com/reese/skills" +PINNED_BRANCH = "main" + +SKILLS_DOC = """# Phased execution for pi.dev + +How does the phased execution process work for pi.dev? It runs one +task at a time: each task file executes in a separate pi subprocess, +and the harness manages phase and task files, retries, and final +validation. + +## Task files + +Every task file is a unit of work. The pipeline runs the todo phases +in numeric order, and a completed phase moves to the complete +directory with its own test suite green. + +## Validation + +A phase is complete only when the unit and integration tests pass, +the coverage gate is green, and the Playwright story suite passes +in isolation. The harness then moves the task file and commits the +phase atomically. +""" + +CONTROL_DOC = """# Control notes + +This is a plain local directory source. It is walked directly by +the importer and never cloned from a repository. The control note +exists so the byte-identity guard has a document: local documents +read, render, and cite exactly as before, with nothing new in the +read result or the meta row. +""" + +# The suite's own content pins — the mock's echo assertion keys off +# the read result's ``origin:``/``branch:`` lines, so neither +# fixture may contain a line with either prefix (a fixture defect, +# not a mock defect — see the mock's ``_origin_link_url``). +assert "origin:" not in SKILLS_DOC and "branch:" not in SKILLS_DOC +assert "origin:" not in CONTROL_DOC and "branch:" not in CONTROL_DOC + +# The scripted questions — the mock's ``ORIGIN_LINK_TRIGGER`` + the +# question-carried call (``_ORIGIN_LINK_CALL_RE``) + the subject +# words (the mock-calibrated gate: both questions stay GROUNDED on +# their own fixture document). +Q_SKILLS = ( + f"Original source link: read {SKILLS_SP} — " + "how does the phased execution process work for pi.dev?" +) +Q_CONTROL = ( + f"Original source link: read {CONTROL_SP} — " + "what does the control note say about plain local sources?" +) + +#: The final answer the mock scripts for the git-matched document +#: (the ``ORIGIN_LINK_ANSWER`` constant with the seeded URL — the +#: model-side contract, imported so the mock can never drift). +ANSWER_SENTENCE = ORIGIN_LINK_ANSWER.format(url=ORIGIN_URL) + + +# -------------------------------------------------------------------------- +# Fixtures +# -------------------------------------------------------------------------- + + +@pytest.fixture(scope="module") +def kb_dirs(tmp_path_factory: pytest.TempPathFactory) -> tuple[Path, Path]: + """The story's two local-directory sources (host temp dirs — the + app server runs on the same host, so the paths are visible to + it): the ``skills`` source (its basename MATCHES the seeded git + row's URL basename — the resolver's name mapping) and the + ``control`` source (no git row — the byte-identity control).""" + skills = tmp_path_factory.mktemp("bor_origin_skills") / SOURCE + skills.mkdir() + (skills / SKILLS_REL).write_text(SKILLS_DOC, encoding="utf-8") + control = tmp_path_factory.mktemp("bor_origin_control") / CONTROL + control.mkdir() + (control / CONTROL_REL).write_text(CONTROL_DOC, encoding="utf-8") + return skills, control + + +def _run_in_thread(coro: Any) -> Any: + """Run a coroutine on a worker thread (the + ``test_document_viewer`` pattern — Playwright's sync API keeps + an asyncio loop on the test thread, so ``asyncio.run`` cannot + run directly in a test body).""" + box: dict[str, Any] = {} + + def runner() -> None: + try: + box["value"] = asyncio.run(coro) + except BaseException as e: # noqa: BLE001 — re-raised in the test + box["error"] = e + + t = Thread(target=runner) + t.start() + t.join() + if "error" in box: + raise box["error"] + return box["value"] + + +async def _import_sources(roots: list[Path], mock_port: int) -> None: + """The house per-module import (the ``test_document_viewer`` + pattern): the REAL importer against the fixture roots under the + deterministic mock embeddings — the mock LLM supplies the + embeds, and the owner's local ``.env`` is NOT read (the + ``_env_file=None`` isolation).""" + kwargs: dict[str, Any] = { + "_env_file": None, + "llm_base_url": f"http://127.0.0.1:{mock_port}/v1", + } + settings = Settings(**kwargs) # pyright: ignore[reportCallIssue] + await import_sources(roots, LLMClient(settings)) + + +def _truncate_all() -> None: + """Fresh registry + KB (the E2E isolation pattern): the E2E + suites share one Postgres, so a leftover git_sources row or + document would pollute the resolver and the retrieval the + scripted turns run against (the two-source KB's gate margins + are pinned against EXACTLY these two documents).""" + with SessionLocal() as db: + db.execute( + text( + "TRUNCATE chunks, documents, query_log, steering_notes, " + "kb_overview, git_sources, folder_summaries" + ) + ) + db.commit() + + +@pytest.fixture(scope="module") +def seeded_kb(mock_llm: int, kb_dirs: tuple[Path, Path]) -> None: + """The story's precondition: the two-source KB (one readable + markdown doc each) imported under the deterministic mock, the + stored content byte-pinned. The git row is NOT part of the + import (a git row present at import time would be cloned — the + row is seeded in the suite's own DB session per test, AFTER + the import; the resolver reads the table live at + read/content time).""" + _truncate_all() + _run_in_thread(_import_sources(list(kb_dirs), mock_llm)) + with SessionLocal() as db: + docs = { + (d.source, d.path): d for d in db.scalars(select(Document)).all() + } + assert set(docs) == {(SOURCE, SKILLS_REL), (CONTROL, CONTROL_REL)}, ( + docs.keys() + ) + assert docs[(SOURCE, SKILLS_REL)].content == SKILLS_DOC + assert docs[(CONTROL, CONTROL_REL)].content == CONTROL_DOC + # The import ran with NO git rows (the local-only KB): the + # seeded row lands after, per test. + assert db.scalar(select(func.count()).select_from(GitSource)) == 0 + + +def _seed_git_row(branch: str | None, exist: bool = True) -> None: + """The git row in the suite's OWN DB session (the task's seam): + delete every row, then — when *exist* — insert the one + ``kind='git'`` row whose URL basename matches the ``skills`` + fixture source (public URL, no credential; *branch* pinned or + NULL). The resolver reads the table LIVE — the row never + touches the importer.""" + with SessionLocal() as db: + db.execute(delete(GitSource)) + if exist: + db.add(GitSource(url=ORIGIN_URL, kind="git", branch=branch)) + db.commit() + + +@pytest.fixture(autouse=True) +def _clean(db_ready: None) -> Iterator[None]: + """Per-test query_log isolation (the KB itself is module-scoped + — the scripted turns never change it, so the documents persist + across the tests of this module).""" + with SessionLocal() as db: + db.execute(text("TRUNCATE query_log")) + db.commit() + yield + with SessionLocal() as db: + db.execute(text("TRUNCATE query_log")) + db.commit() + + +# -------------------------------------------------------------------------- +# Page helpers (the test_read_line_ranges / test_document_viewer +# house patterns) +# -------------------------------------------------------------------------- + +#: Captures the raw SSE ``data:`` payloads of the /api/chat stream +#: (a response clone read in the background) — wire-level +#: assertions for the ``tool`` / ``done`` frames, independent of +#: the UI rendering. +SSE_HOOK = """ +() => { + if (window.__sseInstalled) return; + window.__sseInstalled = true; + window.__sseFrames = []; + const origFetch = window.fetch; + window.fetch = async function (...args) { + const res = await origFetch.apply(this, args); + try { + const url = typeof args[0] === 'string' ? args[0] : args[0].url; + if (url.includes('/api/chat')) { + res.clone().text().then((bodyText) => { + for (const block of bodyText.split('\\n\\n')) { + const line = block.trim(); + if (line.startsWith('data: ')) { + window.__sseFrames.push(line.slice(6)); + } + } + }); + } + } catch (e) { /* non-clonable responses: ignored */ } + return res; + }; +} +""" + + +def _install_page_hooks(page: Page) -> None: + page.evaluate(SSE_HOOK) + + +def _frames(page: Page, deadline_s: float = 120.0) -> list[dict]: + """The SSE frames captured since the last submit (``_submit`` + clears the buffer), once the hook's background read settles.""" + deadline = time.monotonic() + deadline_s + while True: + raw = page.evaluate("() => window.__sseFrames || []") + parsed = [json.loads(line) for line in raw if line] + if any(f.get("type") == "done" for f in parsed): + return parsed + if time.monotonic() > deadline: + raise AssertionError( + f"SSE hook captured no `done` frame (frames so far: " + f"{len(parsed)}) — hook install failed?" + ) + time.sleep(0.05) + + +def _tool_frames(frames: list[dict]) -> list[dict]: + return [f for f in frames if f.get("type") == "tool"] + + +def _submit(page: Page, question: str) -> None: + page.evaluate("window.__sseFrames = []") + page.fill("#message-input", question) + page.click("#send-btn") + # The user bubble lands synchronously with the submit handler. + expect(page.locator(".msg.user .bubble").last).to_contain_text(question) + + +def _wait_settled(page: Page, timeout_ms: int = 120_000) -> None: + """The turn is complete: answer text in the bubble, button + recovered (the phase-48 settle wait).""" + expect(page.locator(".msg.brain .bubble").last).not_to_have_text( + "", timeout=timeout_ms + ) + expect(page.locator("#send-btn")).to_be_enabled(timeout=timeout_ms) + expect(page.locator("#send-label")).to_have_text("Send", timeout=timeout_ms) + + +def _bubble_text(page: Page) -> str: + """The turn's rendered answer (the brain bubble's text content + — the mock's scripted echo lands here: the house scripted-turn + lens on the read result's ``origin:``/``branch:`` lines).""" + return page.locator(".msg.brain .bubble").last.text_content() or "" + + +def _open_modal_for_chip(page: Page, chip_text: str) -> None: + """Drive the story's modal surface: the turn's cited chip + (phase 119 A1 — done.sources is the READ docs, so the read + document is the turn's ONLY chip) opens the same-page document + modal (the ``test_document_viewer`` contract).""" + chip = page.locator(".msg.brain .source-chip", has_text=chip_text) + expect(chip).to_have_count(1, timeout=30_000) + chip.first.click() + expect(page.locator("#doc-modal")).not_to_have_attribute("hidden") + expect(page.locator("#doc-modal")).to_be_visible() + + +# -------------------------------------------------------------------------- +# 1. The answer carries the repository link (the model-side +# contract, through the real read result's ``origin:`` line) +# -------------------------------------------------------------------------- + + +def test_answer_carries_origin_link( + page: Page, app_url: str, seeded_kb: None, db_ready: None +) -> None: + _seed_git_row(branch=None) + page.set_default_timeout(60_000) + login(page, app_url, next="/") + _install_page_hooks(page) + + _submit(page, Q_SKILLS) + _wait_settled(page) + + frames = _frames(page) + # The scripted read on the wire — the turn's only tool call. + assert _tool_frames(frames) == [ + {"type": "tool", "name": "read", "argument": SKILLS_SP} + ], _tool_frames(frames) + done = next(f for f in frames if f.get("type") == "done") + assert done["deflected"] is False, done + # The read document is the turn's cited source (the chip the + # modal tests click — phase 119 A1). + assert any( + s["source"] == SOURCE and s["path"] == SKILLS_REL for s in done["sources"] + ), done["sources"] + + # The read result's surface (the mock's verbatim echo in the + # answer — the house scripted-turn lens): the sanitized + # ``origin:`` line the SERVER put in the result the model saw + # (the task-01 resolver's output — the git row's URL, + # credential-free). + text = _bubble_text(page) + assert f"origin: {ORIGIN_URL}" in text, text + # The done answer carries the repository URL (the scripted + # echo — the model-side contract, deterministic): the mock + # parsed the URL from the origin line — it cannot appear in + # the answer unless the server sent it. + assert ANSWER_SENTENCE in text, text + + +# -------------------------------------------------------------------------- +# 2. The modal's meta row shows the "Original" external anchor +# -------------------------------------------------------------------------- + + +def test_modal_shows_original_anchor( + page: Page, app_url: str, seeded_kb: None, db_ready: None +) -> None: + _seed_git_row(branch=None) + page.set_default_timeout(60_000) + login(page, app_url, next="/") + _install_page_hooks(page) + + _submit(page, Q_SKILLS) + _wait_settled(page) + _open_modal_for_chip(page, SKILLS_REL) + expect(page.locator("#doc-modal-title")).to_have_text( + "Phased execution for pi.dev" + ) + + # The meta row's "Original" anchor — the task-03 contract + # (ONE shared renderDocument site: the modal AND + # /document.html show it). + anchor = page.locator("#doc-modal-meta .doc-origin") + expect(anchor).to_have_count(1) + expect(anchor).to_have_attribute("href", ORIGIN_URL) + expect(anchor).to_have_attribute("target", "_blank") + expect(anchor).to_have_attribute("rel", "noopener") + # The copyable-link contract: the full URL VERBATIM as the + # visible text — NO branch in the text (branch=None), and the + # title is the bare URL (the pathCode precision idiom, no + # branch suffix). + expect(anchor).to_have_text(ORIGIN_URL) + expect(anchor).to_have_attribute("title", ORIGIN_URL) + + +# -------------------------------------------------------------------------- +# 3. The pinned branch rides the read result and the anchor's title +# -------------------------------------------------------------------------- + + +def test_pinned_branch_in_title( + page: Page, app_url: str, seeded_kb: None, db_ready: None +) -> None: + _seed_git_row(branch=PINNED_BRANCH) + page.set_default_timeout(60_000) + login(page, app_url, next="/") + _install_page_hooks(page) + + _submit(page, Q_SKILLS) + _wait_settled(page) + + # The read result's surface (the mock's verbatim echo): the + # origin line AND the pinned branch line (the task-01 second + # line, only when the row pins a branch). + text = _bubble_text(page) + assert f"origin: {ORIGIN_URL}" in text, text + assert f"branch: {PINNED_BRANCH}" in text, text + # The done answer still carries the repository URL (tests + # 1–3's model-side contract — the branch never joins the + # URL). + assert ANSWER_SENTENCE in text, text + + _open_modal_for_chip(page, SKILLS_REL) + anchor = page.locator("#doc-modal-meta .doc-origin") + expect(anchor).to_have_count(1) + expect(anchor).to_have_attribute("href", ORIGIN_URL) + # The text stays the bare URL (copyable — the branch never + # joins the anchor text)… + expect(anchor).to_have_text(ORIGIN_URL) + # …the branch rides the title (the task-03 label contract). + expect(anchor).to_have_attribute( + "title", f"{ORIGIN_URL} — branch: {PINNED_BRANCH}" + ) + + +# -------------------------------------------------------------------------- +# 4. The local-source control is byte-identical (no ``origin:`` +# line in the read result, no anchor in the meta row) +# -------------------------------------------------------------------------- + + +def test_local_source_has_no_origin( + page: Page, app_url: str, seeded_kb: None, db_ready: None +) -> None: + _seed_git_row(branch=None, exist=False) # the pre-phase world: NO git rows at all + page.set_default_timeout(60_000) + login(page, app_url, next="/") + _install_page_hooks(page) + + _submit(page, Q_CONTROL) + _wait_settled(page) + + frames = _frames(page) + assert _tool_frames(frames) == [ + {"type": "tool", "name": "read", "argument": CONTROL_SP} + ], _tool_frames(frames) + done = next(f for f in frames if f.get("type") == "done") + assert done["deflected"] is False, done + + # The byte-identity guard on the read result's surface (the + # mock's verbatim echo): the result carried NO origin line — + # the local source's result is byte-identical to pre-phase, + # so the mock's deterministic no-origin sentence answers. + text = _bubble_text(page) + assert ORIGIN_LINK_NO_ORIGIN in text, text + assert "origin:" not in text, text + assert "branch:" not in text, text + + # …and the modal's meta row is the pre-phase badge row: NO + # .doc-origin anchor (the content payload's origin fields are + # absent on the wire — never null). + _open_modal_for_chip(page, CONTROL_REL) + expect(page.locator("#doc-modal-title")).to_have_text("Control notes") + expect(page.locator("#doc-modal-meta .doc-origin")).to_have_count(0) + # The pre-phase badges are untouched (the row still shows + # source · format · path · created · indexed · chunks). + expect(page.locator("#doc-modal-meta .doc-source-badge")).to_have_text( + CONTROL + ) + expect(page.locator("#doc-modal-meta .doc-path")).to_have_text( + CONTROL_REL + ) + expect(page.locator("#doc-modal-meta .doc-created")).not_to_be_empty() diff --git a/tests/integration/test_document_content.py b/tests/integration/test_document_content.py index 14a2a85..07685b0 100644 --- a/tests/integration/test_document_content.py +++ b/tests/integration/test_document_content.py @@ -10,7 +10,13 @@ Uses the real compose Postgres (``db`` fixture) and FastAPI's TestClient: tests and the anonymous 403/401 pins build their own client; * GET: 404 for an unknown (source, path) pair; * GET: 404 for traversal-style ``path`` values (no filesystem access → no - leak). + leak); +* GET: phase 145 (task 03) — the payload's git origin fields: a seeded + ``kind='git'`` row (URL basename = the fixture source) adds + ``origin_url`` (+ ``origin_branch`` for a pinned row); a legacy + embedded-credential row is served sanitized (end-to-end); a plain + local source keeps the pre-phase key set (both keys ABSENT — never + ``null``). PATCH /api/documents/summary (phase 57, task 01) — the admin summary editor, on the same DB-backed fixtures: @@ -27,6 +33,7 @@ editor, on the same DB-backed fixtures: """ from __future__ import annotations +from collections.abc import Iterator from datetime import UTC, datetime import pytest @@ -37,7 +44,7 @@ from sqlalchemy.orm import Session import app.api.docs as docs_api from app.config import get_settings from app.main import app as fastapi_app -from app.models import Chunk, Document +from app.models import Chunk, Document, GitSource from app.rag.llm import EmbeddingError from tests.conftest import ADMIN_PASSWORD from tests.fakes import FakeEmbedder @@ -571,6 +578,131 @@ def test_content_format_from_suffix(client, db) -> None: db.commit() +# --------------------------------------------------------------------------- +# Phase 145 (task 03) — the content payload's git origin fields +# (the task-01 resolver's output, end-to-end over the real registry) +# --------------------------------------------------------------------------- + + +@pytest.fixture() +def origin_registry(db) -> Iterator[None]: + """A FRESH ``git_sources`` table around the origin tests: the + resolver reads EVERY ``kind='git'`` row, so a controlled registry + is required (truncated before and after — the + ``test_agent_tools.py`` registry pattern).""" + db.execute(text("TRUNCATE git_sources")) + db.commit() + yield + db.execute(text("TRUNCATE git_sources")) + db.commit() + + +def test_content_carries_origin_url_for_a_seeded_git_row( + client, db, origin_registry +) -> None: + """A ``kind='git'`` row whose URL basename matches the document's + source → the payload carries ``origin_url`` (the sanitized clone + URL). The row pins no branch (NULL = the default branch) → + ``origin_branch`` is ABSENT from the JSON (not ``null`` — the + omission pinned on the wire); every pre-phase key stands.""" + _seed_doc(db) + db.add(GitSource(url="https://github.com/reese/Homelab.git", kind="git")) + db.commit() + try: + r = client.get( + "/api/documents/content", + params={"source": "Homelab", "path": "kubernetes.md"}, + ) + assert r.status_code == 200 + body = r.json() + assert set(body) == { + "source", "path", "title", "format", "summary", "created_at", + "content", "indexed_at", "chunks", "is_image", "origin_url", + } + assert body["origin_url"] == "https://github.com/reese/Homelab.git" + assert "origin_branch" not in body # absent — never null + finally: + db.execute(text("TRUNCATE chunks, documents")) + db.commit() + + +def test_content_carries_origin_branch_for_a_pinned_row( + client, db, origin_registry +) -> None: + """A pinned row (phase 135 ``branch``) also carries + ``origin_branch`` — the value the viewer's title attribute shows.""" + _seed_doc(db) + db.add( + GitSource( + url="https://github.com/reese/Homelab.git", kind="git", branch="docs" + ) + ) + db.commit() + try: + r = client.get( + "/api/documents/content", + params={"source": "Homelab", "path": "kubernetes.md"}, + ) + body = r.json() + assert body["origin_url"] == "https://github.com/reese/Homelab.git" + assert body["origin_branch"] == "docs" + finally: + db.execute(text("TRUNCATE chunks, documents")) + db.commit() + + +def test_content_origin_is_sanitized_end_to_end(client, db, origin_registry) -> None: + """A legacy row whose credential is still embedded in the stored + URL never leaks it: the payload carries the phase-121 sanitized + URL (the resolver's guarantee, end-to-end — the stored row value + is untouched, the output mask does the work).""" + _seed_doc(db) + db.add( + GitSource(url="https://user:tok@example.com/reese/Homelab.git", kind="git") + ) + db.commit() + try: + r = client.get( + "/api/documents/content", + params={"source": "Homelab", "path": "kubernetes.md"}, + ) + body = r.json() + assert body["origin_url"] == "https://example.com/reese/Homelab.git" + assert "user:tok" not in r.text # the credential stays out of the wire + finally: + db.execute(text("TRUNCATE chunks, documents")) + db.commit() + + +def test_content_origin_absent_for_a_plain_local_source( + client, db, origin_registry +) -> None: + """No unambiguous git row (a local row is never an origin, and a + non-matching git row does not count) → BOTH origin keys ABSENT + from the JSON (not ``null`` — the omission pinned on the wire; the + payload is byte-identical to pre-phase).""" + _seed_doc(db) + db.add(GitSource(url="/srv/Homelab", kind="local", path="/srv/Homelab")) + db.add(GitSource(url="https://github.com/reese/OtherRepo.git", kind="git")) + db.commit() + try: + r = client.get( + "/api/documents/content", + params={"source": "Homelab", "path": "kubernetes.md"}, + ) + assert r.status_code == 200 + body = r.json() + assert set(body) == { + "source", "path", "title", "format", "summary", "created_at", + "content", "indexed_at", "chunks", "is_image", + } + assert "origin_url" not in body + assert "origin_branch" not in body + finally: + db.execute(text("TRUNCATE chunks, documents")) + db.commit() + + def test_content_format_name_token_extensionless( client, db, monkeypatch ) -> None: diff --git a/tests/unit/test_agent.py b/tests/unit/test_agent.py index 9e872ee..872e08b 100644 --- a/tests/unit/test_agent.py +++ b/tests/unit/test_agent.py @@ -269,8 +269,11 @@ def test_agent_tools_names_and_parameters() -> None: "and `limit` (number of lines) to read exactly that " "part. Do not re-read a document you have already read " "in full — its full text is already in your prompt. " - "Call one tool at a time — wait for this result before " - "your next call." + "For a document from a git source, the result also " + "carries an `origin` line (and `branch` when pinned) — " + "include that original source in your answer (see your " + "tools instructions). Call one tool at a time — wait " + "for this result before your next call." ) # The byte-preserved contracts inside the rewrite, pinned as # substrings: the combined source/path identity, the @@ -293,6 +296,17 @@ def test_agent_tools_names_and_parameters() -> None: "Call one tool at a time — wait for this result before " "your next call." ) in read["description"] + # Phase 145 (task 02): the origin clause mirrors the TOOLS_SECTION + # rule (the description points, the section teaches — the + # phase-124 show_image "see the instructions" shape): a git- + # sourced document's read result carries an `origin` line (and + # `branch` when pinned); include that original source in the + # answer. + assert ( + "For a document from a git source, the result also carries " + "an `origin` line (and `branch` when pinned) — include that " + "original source in your answer (see your tools instructions)." + ) in read["description"] read_params = read["parameters"] assert read_params["type"] == "object" # Phase 131 (task 01, LOCKED A2): offset/limit are OPTIONAL — @@ -3649,6 +3663,320 @@ def test_unknown_tool_name_refused(monkeypatch: pytest.MonkeyPatch) -> None: assert llm.requests[1][1] == AGENT_TOOLS # rejected → tools stay offered +# ---------- phase 145 (task 01): the origin/branch lines in the read result ---------- + +#: The fixture git row's URL (basename ``skills`` = the fixture source +#: name) and the four origin outcomes the read-branch matrix runs: +#: git row matched (branch NULL), git row + pinned branch, local source +#: (no git row), two rows sharing the basename (ambiguous — the source +#: name is ``repo`` for the last one). +_GIT_URL = "https://github.com/owner/skills" +_GIT_ROWS = [GitSource(url=_GIT_URL, kind="git")] +_GIT_PINNED_ROWS = [GitSource(url=_GIT_URL, kind="git", branch="release/1.2")] +_LOCAL_ROWS = [GitSource(url="/srv/skills", kind="local", path="/srv/skills")] +_AMBIGUOUS_ROWS = [ + GitSource(url="https://a.example/org/repo.git", kind="git"), + GitSource(url="git@b.example:other/repo.git", kind="git"), +] + + +def _git_rows_factory(rows: list[GitSource]) -> Callable[[], Session]: + """A db factory whose mock session answers the resolver's query with + the git rows of *rows* (like the real ``kind = 'git'`` WHERE clause + — the seeded local rows are filtered out, proving they are never + candidates): the REAL ``origin_for_source`` runs in the read branch, + no database.""" + git_rows = [row for row in rows if row.kind == "git"] + + def _factory() -> Session: + mock = cast("Session", MagicMock()) + mock.__enter__ = MagicMock(return_value=mock) + mock.__exit__ = MagicMock(return_value=False) + mock.scalars = MagicMock(return_value=MagicMock(all=MagicMock(return_value=git_rows))) + return mock + + return _factory + + +def _one_read( + monkeypatch: pytest.MonkeyPatch, + doc: Document, + rows: list[GitSource], + arguments: dict[str, Any], + read_max_chars: int | None = None, +) -> str: + """One scripted read turn against the REAL resolver; returns the + exact tool-result string (the successful-read bookkeeping is + asserted — the matrix asserts the bytes).""" + monkeypatch.setattr(agent, "find_document", lambda db, source, path: doc) + holder = AgentHolder() + llm = ScriptedLLM( + [ToolCallPiece(id="c1", name="read", arguments=arguments)], + [StreamPiece("content", "ans")], + ) + kwargs: dict[str, Any] = {"read_max_chars": read_max_chars} if read_max_chars else {} + asyncio.run(_run(llm, holder, _settings(**kwargs), db_factory=_git_rows_factory(rows))) + assert holder.read_docs == [doc] and holder.tool_calls == 1 + return llm.requests[1][0][3]["content"] + + +def test_origin_lines_plain_read_under_cap( + monkeypatch: pytest.MonkeyPatch, caplog: pytest.LogCaptureFixture +) -> None: + """Shape 1 (plain read at or under the cap): the origin block rides + IMMEDIATELY AFTER the ``date:`` line — ``Document …: / date: / + origin: / branch: / content`` — in all four origin outcomes; the + first line (the mock's ``_READ_RESULT_PREFIX`` contract) is + untouched, and the local/ambiguous results are byte-identical to + pre-phase (the ambiguity still warns).""" + doc = _doc("skills", "docs/a.md", "A", "A-CONTENT") + plain = {"path": "skills/docs/a.md"} + + result = _one_read(monkeypatch, doc, _GIT_ROWS, plain) + assert result == ( + "Document skills/docs/a.md:\n" + "date: 2024-06-15\n" + f"origin: {_GIT_URL}\n" + "A-CONTENT" + ) + # The E2E mock's header contract: first line, byte-identical. + assert result.startswith("Document ") + assert result.splitlines()[0] == "Document skills/docs/a.md:" + + result = _one_read(monkeypatch, doc, _GIT_PINNED_ROWS, plain) + assert result == ( + "Document skills/docs/a.md:\n" + "date: 2024-06-15\n" + f"origin: {_GIT_URL}\n" + "branch: release/1.2\n" + "A-CONTENT" + ) + + result = _one_read(monkeypatch, doc, _LOCAL_ROWS, plain) + assert result == "Document skills/docs/a.md:\ndate: 2024-06-15\nA-CONTENT" + + repo_doc = _doc("repo", "docs/a.md", "A", "A-CONTENT") + caplog.clear() + result = _one_read(monkeypatch, repo_doc, _AMBIGUOUS_ROWS, {"path": "repo/docs/a.md"}) + assert result == "Document repo/docs/a.md:\ndate: 2024-06-15\nA-CONTENT" + # The ambiguity is logged at WARNING (the agent's INFO per-turn log + # lines are captured too — filter to the resolver's records). + warnings = [ + r for r in caplog.records if r.levelname == "WARNING" and r.name == "app.rag.git_sources" + ] + assert len(warnings) == 1 + assert "repo" in warnings[0].message + + +def test_origin_lines_truncated_plain_read( + monkeypatch: pytest.MonkeyPatch, +) -> None: + """Shape 2 (plain read cut at the cap): the origin block sits at the + same position (after ``date:``, before the cut content), the + truncation marker + the pinned notice unchanged, the holder + truncation tuple unchanged (the ``read_truncations`` signal is + untouched).""" + cap = 20 + content = "x" * (cap + 1) + doc = _doc("skills", "big.md", "Big", content) + plain = {"path": "skills/big.md"} + + result = _one_read(monkeypatch, doc, _GIT_ROWS, plain, read_max_chars=cap) + assert result == ( + "Document skills/big.md:\n" + "date: 2024-06-15\n" + f"origin: {_GIT_URL}\n" + + content[:cap] + + "\n" + + TRUNCATION_MARKER + + "\n" + + READ_TRUNCATION_NOTICE.format(shown=cap, total=len(content)) + ) + + result = _one_read(monkeypatch, doc, _GIT_PINNED_ROWS, plain, read_max_chars=cap) + assert result == ( + "Document skills/big.md:\n" + "date: 2024-06-15\n" + f"origin: {_GIT_URL}\n" + "branch: release/1.2\n" + + content[:cap] + + "\n" + + TRUNCATION_MARKER + + "\n" + + READ_TRUNCATION_NOTICE.format(shown=cap, total=len(content)) + ) + + result = _one_read(monkeypatch, doc, _LOCAL_ROWS, plain, read_max_chars=cap) + assert result == ( + "Document skills/big.md:\n" + "date: 2024-06-15\n" + + content[:cap] + + "\n" + + TRUNCATION_MARKER + + "\n" + + READ_TRUNCATION_NOTICE.format(shown=cap, total=len(content)) + ) + + repo_doc = _doc("repo", "big.md", "Big", content) + result = _one_read( + monkeypatch, repo_doc, _AMBIGUOUS_ROWS, {"path": "repo/big.md"}, read_max_chars=cap + ) + assert result == ( + "Document repo/big.md:\n" + "date: 2024-06-15\n" + + content[:cap] + + "\n" + + TRUNCATION_MARKER + + "\n" + + READ_TRUNCATION_NOTICE.format(shown=cap, total=len(content)) + ) + + +def test_origin_lines_range_read_under_cap( + monkeypatch: pytest.MonkeyPatch, caplog: pytest.LogCaptureFixture +) -> None: + """Shape 3 (range read at or under the cap): the origin block rides + between the ``date:`` line and the ``lines:`` header (the same + position as in the plain shapes); the local/ambiguous results stay + byte-identical to pre-phase.""" + doc = _doc("skills", "b.md", "B", "one\ntwo\nthree\nfour\nfive") + ranged = {"path": "skills/b.md", "offset": 2, "limit": 2} + + result = _one_read(monkeypatch, doc, _GIT_ROWS, ranged) + assert result == ( + "Document skills/b.md:\n" + "date: 2024-06-15\n" + f"origin: {_GIT_URL}\n" + "lines: 2-3 of 5\n" + "two\nthree" + ) + + result = _one_read(monkeypatch, doc, _GIT_PINNED_ROWS, ranged) + assert result == ( + "Document skills/b.md:\n" + "date: 2024-06-15\n" + f"origin: {_GIT_URL}\n" + "branch: release/1.2\n" + "lines: 2-3 of 5\n" + "two\nthree" + ) + + result = _one_read(monkeypatch, doc, _LOCAL_ROWS, ranged) + assert result == "Document skills/b.md:\ndate: 2024-06-15\nlines: 2-3 of 5\ntwo\nthree" + + repo_doc = _doc("repo", "b.md", "B", "one\ntwo\nthree\nfour\nfive") + caplog.clear() + result = _one_read( + monkeypatch, repo_doc, _AMBIGUOUS_ROWS, {"path": "repo/b.md", "offset": 2, "limit": 2} + ) + assert result == "Document repo/b.md:\ndate: 2024-06-15\nlines: 2-3 of 5\ntwo\nthree" + # The ambiguity is logged at WARNING (the agent's INFO per-turn log + # lines are captured too — filter to the resolver's records). + warnings = [ + r for r in caplog.records if r.levelname == "WARNING" and r.name == "app.rag.git_sources" + ] + assert len(warnings) == 1 + assert "repo" in warnings[0].message + + +def test_origin_lines_truncated_range_read( + monkeypatch: pytest.MonkeyPatch, +) -> None: + """Shape 4 (range read whose SLICE exceeds the cap): the origin block + at the same position; the cap on the slice + the shared marker + the + pinned notice (``{total}`` = the SLICE's length) all unchanged.""" + cap = 40 + doc = _doc("skills", "c.md", "C", "a" * 30 + "\n" + "b" * 30 + "\n" + "c" * 30) + slice_text = "a" * 30 + "\n" + "b" * 30 # the requested lines 1-2 (61 chars) + ranged = {"path": "skills/c.md", "offset": 1, "limit": 2} + + result = _one_read(monkeypatch, doc, _GIT_ROWS, ranged, read_max_chars=cap) + assert result == ( + "Document skills/c.md:\n" + "date: 2024-06-15\n" + f"origin: {_GIT_URL}\n" + "lines: 1-2 of 3\n" + + slice_text[:cap] + + "\n" + + TRUNCATION_MARKER + + "\n" + + READ_TRUNCATION_NOTICE.format(shown=cap, total=len(slice_text)) + ) + + result = _one_read(monkeypatch, doc, _GIT_PINNED_ROWS, ranged, read_max_chars=cap) + assert result == ( + "Document skills/c.md:\n" + "date: 2024-06-15\n" + f"origin: {_GIT_URL}\n" + "branch: release/1.2\n" + "lines: 1-2 of 3\n" + + slice_text[:cap] + + "\n" + + TRUNCATION_MARKER + + "\n" + + READ_TRUNCATION_NOTICE.format(shown=cap, total=len(slice_text)) + ) + + result = _one_read(monkeypatch, doc, _LOCAL_ROWS, ranged, read_max_chars=cap) + assert result == ( + "Document skills/c.md:\n" + "date: 2024-06-15\n" + "lines: 1-2 of 3\n" + + slice_text[:cap] + + "\n" + + TRUNCATION_MARKER + + "\n" + + READ_TRUNCATION_NOTICE.format(shown=cap, total=len(slice_text)) + ) + + repo_doc = _doc("repo", "c.md", "C", "a" * 30 + "\n" + "b" * 30 + "\n" + "c" * 30) + result = _one_read( + monkeypatch, repo_doc, _AMBIGUOUS_ROWS, {"path": "repo/c.md", "offset": 1, "limit": 2}, + read_max_chars=cap, + ) + assert result == ( + "Document repo/c.md:\n" + "date: 2024-06-15\n" + "lines: 1-2 of 3\n" + + slice_text[:cap] + + "\n" + + TRUNCATION_MARKER + + "\n" + + READ_TRUNCATION_NOTICE.format(shown=cap, total=len(slice_text)) + ) + + +def test_origin_lines_image_doc_keeps_marker_position( + monkeypatch: pytest.MonkeyPatch, +) -> None: + """The image-doc shape keeps its phase-122 pin: the marker stays + IMMEDIATELY ABOVE the content, and the origin lines ride ABOVE the + marker (``Document …: / date: / origin: / branch: / marker / + content``); a local image doc is byte-identical to pre-phase.""" + doc = _doc("skills", "img.png", "Img", "IMG-DESC") + doc.is_image = True + plain = {"path": "skills/img.png"} + + result = _one_read(monkeypatch, doc, _GIT_PINNED_ROWS, plain) + assert result == ( + "Document skills/img.png:\n" + "date: 2024-06-15\n" + f"origin: {_GIT_URL}\n" + "branch: release/1.2\n" + f"{agent.IMAGE_DOC_MARKER}\n" + "IMG-DESC" + ) + + doc = _doc("skills", "img.png", "Img", "IMG-DESC") + doc.is_image = True + result = _one_read(monkeypatch, doc, _LOCAL_ROWS, plain) + assert result == ( + "Document skills/img.png:\ndate: 2024-06-15\n" + f"{agent.IMAGE_DOC_MARKER}\n" + "IMG-DESC" + ) + + # ---------- grep (the phase-68 A5 contract under the new name) ---------- diff --git a/tests/unit/test_document_viewer.py b/tests/unit/test_document_viewer.py index d9eabf4..3585f2d 100644 --- a/tests/unit/test_document_viewer.py +++ b/tests/unit/test_document_viewer.py @@ -20,9 +20,29 @@ Frontend side: * phase 26 — the shared ``renderDocument`` export in ``document.js``, the import-safe page guard, the modal skeleton on chat + Sources, and the modal module's close/focus/URL contract. + +Phase 145 (task 03) — the content payload's git origin fields + the +``renderDocument`` "Original" anchor (modal AND /document.html, one +site): +* the ``DocContent`` omission rule — both origin keys ABSENT on the + wire when ``None`` (never ``null``; non-git payload byte-identical); +* the content endpoint (stubbed session) — a matching ``kind='git'`` + row adds ``origin_url`` (+ ``origin_branch`` for a pinned row), a + legacy embedded-credential row is served sanitized, and local / + ambiguous sources keep the pre-phase key set; +* the anchor itself (source pins + a node behavioral check over a + minimal fake DOM) — ``createElement("a")``, the full URL verbatim as + ``href`` AND visible text (the copyable-link contract), + ``target="_blank" rel="noopener"``, the ``title`` = full URL (+ + `` — branch: `` when pinned), inserted AFTER the path code and + BEFORE the Created badge (absent → the pre-phase six children), no + innerHTML/outerHTML (the module's ONE innerHTML survives), and the + ``.doc-origin`` CSS mirroring the ``.doc-path`` ellipsis idiom with + the ``.doc-link`` affordance (+ the ≤640px squeeze). """ from __future__ import annotations +import json import re import shutil import subprocess @@ -35,7 +55,8 @@ from fastapi.testclient import TestClient from app.api.docs import doc_format from app.db import get_db from app.main import create_app -from app.models import Document +from app.models import Document, GitSource +from app.schemas import DocContent from tests.conftest import ADMIN_PASSWORD FRONTEND = Path(__file__).resolve().parents[2] / "frontend" @@ -126,22 +147,39 @@ class _FakeResult: return self._row +class _FakeScalars: + """The resolver's ``db.scalars(...).all()`` — only the git rows + ever reach the caller (the ``kind = 'git'`` WHERE clause is + emulated, the task-01 ``_GitOnlySession`` pattern).""" + + def __init__(self, rows: tuple[GitSource, ...]) -> None: + self._rows = [row for row in rows if row.kind == "git"] + + def all(self) -> list[GitSource]: + return self._rows + + class _FakeSession: - def __init__(self, row: object) -> None: + def __init__(self, row: object, git_rows: tuple[GitSource, ...] = ()) -> None: self._row = row + self._git_rows = git_rows def execute(self, _stmt: object) -> _FakeResult: return _FakeResult(self._row) + def scalars(self, _stmt: object) -> _FakeScalars: + return _FakeScalars(self._git_rows) -def _client_with_row(row: object) -> TestClient: + +def _client_with_row(row: object, git_rows: tuple[GitSource, ...] = ()) -> TestClient: """Fresh app whose ``get_db`` dependency is a stub returning ``row`` - (``None`` → no matching document row). Phase 79: the endpoint is - user-gated, so the client signs in as the admin first — these tests - pin the CONTENT mapping (200/404/422), not the auth contract (which - ``test_auth_api.py`` pins).""" + (``None`` → no matching document row) and, for the phase-145 + origin lookup, *git_rows* (default: no rows — the non-git corner). + Phase 79: the endpoint is user-gated, so the client signs in as + the admin first — these tests pin the CONTENT mapping (200/404/ + 422), not the auth contract (which ``test_auth_api.py`` pins).""" app = create_app() - app.dependency_overrides[get_db] = lambda: _FakeSession(row) + app.dependency_overrides[get_db] = lambda: _FakeSession(row, git_rows) client = TestClient(app) r = client.post("/api/login", json={"password": ADMIN_PASSWORD}) assert r.status_code == 204, f"admin login failed: {r.status_code} {r.text}" @@ -217,6 +255,322 @@ def test_content_requires_both_params() -> None: assert client.get("/api/documents/content", params={"source": "Homelab"}).status_code == 422 +# --------------------------------------------------------------------------- +# Phase 145 (task 03) — the content payload's git origin fields + the +# renderDocument "Original" anchor (modal AND /document.html, one site) +# --------------------------------------------------------------------------- + + +def _text_doc() -> Document: + """The content-endpoint stub's document row (a markdown doc — no + summary, the pre-phase shape).""" + doc = Document( + source="Homelab", + path="notes/deep mark.md", + full_path="/tmp/deep mark.md", + title="Deep Mark", + content="# Deep Mark\n\nbody", + content_hash="f" * 64, + is_image=False, # phase 122: the stub session never applies defaults + ) + doc.indexed_at = datetime(2026, 8, 22, 1, 2, 3, tzinfo=UTC) + doc.created_at = datetime(2026, 8, 20, 9, 0, 0, tzinfo=UTC) + return doc + + +def _text_doc_content( + origin_url: str | None = None, origin_branch: str | None = None +) -> DocContent: + """One text doc's content payload (the pre-phase shape) with the + phase-145 origin fields optionally set.""" + return DocContent( + source="S", + path="a.md", + title="A", + format="md", + created_at="2026-01-01T00:00:00+00:00", + content="body", + indexed_at="2026-01-01T00:00:00+00:00", + chunks=1, + origin_url=origin_url, + origin_branch=origin_branch, + ) + + +def test_doc_content_origin_fields_absent_on_the_wire_when_unset() -> None: + """The model-level omission rule (the :func:`_drop_absent_share_url` + precedent, the ``image_url`` shape): a non-git document (no + ``origin_url``) serializes with BOTH origin keys ABSENT — never + ``null`` — so the payload is byte-identical to pre-phase. A + sanitized-URL-only payload (an unpinned row) drops + ``origin_branch``; a pinned row keeps both keys.""" + dumped = _text_doc_content().model_dump() + assert "origin_url" not in dumped # absent — never null + assert "origin_branch" not in dumped # absent — never null + dumped = _text_doc_content(origin_url="https://host/x.git").model_dump() + assert dumped["origin_url"] == "https://host/x.git" + assert "origin_branch" not in dumped # NULL = the default branch — nothing to show + dumped = _text_doc_content( + origin_url="https://host/x.git", origin_branch="docs" + ).model_dump() + assert dumped["origin_branch"] == "docs" + + +def test_content_origin_url_carried_for_a_matching_git_row() -> None: + """The endpoint (stubbed session, no database): a ``kind='git'`` + row whose URL basename matches the document's source adds + ``origin_url`` — the sanitized clone URL — to the payload, and + every pre-phase key stands untouched. The row pins no branch → + ``origin_branch`` is ABSENT (never ``null``).""" + row = GitSource(url="https://github.com/reese/Homelab.git", kind="git") + with _client_with_row((_text_doc(), 3), git_rows=(row,)) as client: + r = client.get( + "/api/documents/content", + params={"source": "Homelab", "path": "notes/deep mark.md"}, + ) + assert r.status_code == 200 + body = r.json() + assert set(body) == { + "source", "path", "title", "format", "summary", "created_at", + "content", "indexed_at", "chunks", "is_image", "origin_url", + } + assert body["origin_url"] == "https://github.com/reese/Homelab.git" + assert "origin_branch" not in body # absent — never null (unpinned row) + + +def test_content_origin_branch_carried_for_a_pinned_row() -> None: + """A pinned row (phase 135 ``branch``) also carries + ``origin_branch`` — the branch the title attribute will show.""" + row = GitSource( + url="https://github.com/reese/Homelab.git", kind="git", branch="docs" + ) + with _client_with_row((_text_doc(), 3), git_rows=(row,)) as client: + r = client.get( + "/api/documents/content", + params={"source": "Homelab", "path": "notes/deep mark.md"}, + ) + body = r.json() + assert body["origin_url"] == "https://github.com/reese/Homelab.git" + assert body["origin_branch"] == "docs" + + +def test_content_origin_url_sanitizes_a_legacy_embedded_credential_row() -> None: + """End-to-end (stubbed): a legacy row whose credential is still + embedded in the stored URL never leaks it — the payload carries + the phase-121 sanitized URL (the resolver's guarantee, one mapping + for the whole app).""" + row = GitSource(url="https://user:tok@example.com/reese/Homelab.git", kind="git") + with _client_with_row((_text_doc(), 3), git_rows=(row,)) as client: + r = client.get( + "/api/documents/content", + params={"source": "Homelab", "path": "notes/deep mark.md"}, + ) + body = r.json() + assert body["origin_url"] == "https://example.com/reese/Homelab.git" + assert "user:tok" not in r.text # the credential stays out of the wire + + +def test_content_origin_fields_absent_for_local_and_ambiguous_sources() -> None: + """No git row (a plain local source) OR two rows sharing the + basename (ambiguous — never guess) → both origin keys ABSENT from + the JSON (not ``null`` — the omission pinned on the wire; the + payload is byte-identical to pre-phase, the ``test_content_known_ + pair_maps_to_doc_content`` key-set pin).""" + local_only = (_text_doc(), 3) + with _client_with_row(local_only) as client: # no git rows at all + r = client.get( + "/api/documents/content", + params={"source": "Homelab", "path": "notes/deep mark.md"}, + ) + assert r.status_code == 200 + assert "origin_url" not in r.json() + assert "origin_branch" not in r.json() + + ambiguous = ( + GitSource(url="https://a.example/org/Homelab.git", kind="git"), + GitSource(url="git@b.example:other/Homelab.git", kind="git"), + ) + with _client_with_row(local_only, git_rows=ambiguous) as client: + r = client.get( + "/api/documents/content", + params={"source": "Homelab", "path": "notes/deep mark.md"}, + ) + assert r.status_code == 200 + assert "origin_url" not in r.json() + assert "origin_branch" not in r.json() + + +# ---------- the renderDocument "Original" anchor (the shared core) ---------- + + +def _fn(js: str, name: str) -> str: + """The source of a top-level function via balanced-brace counting + (the ``test_date_editor.py`` pattern).""" + start = js.find(f"function {name}(") + assert start != -1, f"{name}() must exist in document.js" + i = js.find("(", start) + depth = 0 + close = i + while i < len(js): + if js[i] == "(": + depth += 1 + elif js[i] == ")": + depth -= 1 + if depth == 0: + close = i + break + i += 1 + brace = js.find("{", close) + depth = 0 + for j in range(brace, len(js)): + if js[j] == "{": + depth += 1 + elif js[j] == "}": + depth -= 1 + if depth == 0: + return js[start : j + 1] + raise AssertionError(f"unbalanced braces in {name}()") + + +def test_origin_anchor_is_a_copyable_external_link() -> None: + """The anchor (house-style source contract): ``createElement("a")`` + (the never-build-HTML rule), the ``doc-origin`` class, ``href = + doc.origin_url``, ``target="_blank" rel="noopener"`` (the modal's + "Full page" escape-hatch idiom), the full URL VERBATIM as the + visible text (the copyable-link contract — no decorative label), + and the ``title`` = the full URL, plus `` — branch: `` when + the payload pins a branch (the pathCode precision idiom).""" + body = _fn(_read(DOCUMENT_JS), "originAnchor") + assert 'document.createElement("a")' in body + assert 'a.className = "doc-origin"' in body + assert "a.href = doc.origin_url" in body + assert 'a.target = "_blank"' in body + assert 'a.rel = "noopener"' in body + assert "a.textContent = doc.origin_url" in body, ( + "the visible text is the full URL verbatim (copy-pasteable)" + ) + branch_title = 'doc.origin_url + " — branch: " + doc.origin_branch' + assert 'doc.origin_branch ? ' + branch_title + " : doc.origin_url" in body, ( + "the title carries the full URL (+ the pinned branch when present)" + ) + # The never-build-HTML shape: no innerHTML/outerHTML in the new code. + code = re.sub(r"//.*?$|/\*.*?\*/", "", body, flags=re.S | re.M) + assert "innerHTML" not in code and "outerHTML" not in code + + +@pytest.mark.skipif(not HAVE_NODE, reason="node not available") +def test_origin_anchor_behaves_for_branch_and_no_branch_variants() -> None: + """Behavioral check (node, minimal fake DOM) of the real anchor + builder: the exact ``href``/``target``/``rel``/``textContent``/ + ``title`` for a git doc, branch and no-branch variants.""" + fn = _fn(_read(DOCUMENT_JS), "originAnchor") + fake_dom = ( + "globalThis.document = { createElement: (tag) => {" + ' const el = { tag, className: "", href: "", target: "", rel: "",' + ' textContent: "", setAttribute(k, v) { this[k] = v; } };' + " return el; } };" + ) + out = _run_node( + fake_dom + + "\n" + fn + + '\nconst g1 = originAnchor({ origin_url: "https://github.com/reese/skills.git",' + ' origin_branch: "main" });' + "\n" + + '\nconst g2 = originAnchor({ origin_url: "https://github.com/reese/skills.git" });' + "\n" + + "console.log(JSON.stringify(g1));\nconsole.log(JSON.stringify(g2));" + ) + j1, j2 = (json.loads(line) for line in out.splitlines()) + for j in (j1, j2): + assert j["tag"] == "a" + assert j["className"] == "doc-origin" + assert j["href"] == "https://github.com/reese/skills.git" + assert j["target"] == "_blank" + assert j["rel"] == "noopener" + # The full URL VERBATIM as the visible text (copy-pasteable). + assert j["textContent"] == "https://github.com/reese/skills.git" + # The title: the full URL, + the pinned branch when present. + assert j1["title"] == "https://github.com/reese/skills.git — branch: main" + assert j2["title"] == "https://github.com/reese/skills.git" + + +def test_origin_anchor_sits_after_the_path_code_before_the_created_badge() -> None: + """renderDocument (the ONE shared core — modal AND /document.html): + the anchor joins the meta row AFTER the path ``code`` and BEFORE + the Created badge — and ONLY when the payload carries + ``origin_url``. Absent (a non-git document — the field simply + missing on the wire) → the gate builds the empty array, the + spread adds nothing, and the meta row's children are EXACTLY the + pre-phase six (the existing badge-order pins pass unchanged).""" + render = _fn(_read(DOCUMENT_JS), "renderDocument") + gate_i = render.find("const originItems = doc.origin_url ? [originAnchor(doc)] : [];") + call_i = render.find("metaEl.replaceChildren(") + path_i = render.find("pathCode,", call_i) + spread_i = render.find("...originItems,", call_i) + created_i = render.find('metaBadge("doc-created"', call_i) + assert -1 < gate_i < call_i < path_i < spread_i < created_i, ( + "the anchor gate precedes the row build and lands AFTER the path code, " + "BEFORE the Created badge" + ) + # The gate: present only on a truthy origin_url — an ABSENT field + # (undefined) is falsy → [] → the pre-phase children stand. + assert "doc.origin_url ? [originAnchor(doc)] : []" in render + # The spread is the meta row's ONLY conditional child (the six + # pre-phase children are the fixed ones). + assert render[call_i:].count("...") == 1 + + +def test_origin_anchor_keeps_the_module_xss_contract() -> None: + """The module's ONE innerHTML (the escape-first markdown render) + survives — the new anchor adds no HTML-building of its own + (createElement + properties/textContent only).""" + code = re.sub(r"//.*", "", re.sub(r"/\*.*?\*/", "", _read(DOCUMENT_JS), flags=re.S)) + assert code.count("innerHTML") == 1 + assert "outerHTML" not in code + + +def test_doc_origin_css_mirrors_the_doc_path_ellipsis_idiom() -> None: + """styles.css: the ``.doc-origin`` rule sits next to the + ``.doc-path`` family (mono, 0.75rem, the same 26rem ellipsis + budget + 6rem floor) with the ``.doc-link`` affordance + (brand-ink, underline on hover/focus — the existing link + tokens), provenance-cited, and the ≤640px block squeezes it like + the path (16rem — the row's own wrap/clip rules are unchanged: + the anchor is one more inline item).""" + css = _read(FRONTEND / "assets" / "styles.css") + rule_i = css.find(".doc-origin {") + assert rule_i != -1, "styles.css must style .doc-origin" + # Provenance comment directly above the rule (house style). + comment_start = css.rfind("/*", 0, rule_i) + assert -1 < comment_start < rule_i and "Phase 145 (task 03)" in css[comment_start:rule_i] + # Next to the .doc-path family (before .doc-chunks). + assert css.find(".doc-path {") < rule_i < css.find(".doc-chunks {") + rule = css[rule_i : css.find("}", rule_i)] + for decl in ( + "font-family: var(--mono)", + "font-size: 0.75rem", + "max-width: 26rem", + "white-space: nowrap", + "overflow: hidden", + "text-overflow: ellipsis", + "min-width: 6rem", + "color: var(--brand-ink)", + "text-decoration: none", + ): + assert decl in rule, f".doc-origin must carry {decl!r}" + # The .doc-link hover/focus affordance (the existing link idiom). + assert ".doc-origin:hover, .doc-origin:focus-visible { text-decoration: underline; }" in css + # The ≤640px squeeze mirrors .doc-path's 16rem (same media block). + mobile = css.find("@media (max-width: 640px)") + path_sq = css.find(".doc-path { max-width: 16rem; }", mobile) + origin_sq = css.find(".doc-origin { max-width: 16rem; }", mobile) + close_640 = css.find("@media", mobile + 1) + assert -1 < path_sq < origin_sq < (close_640 if close_640 != -1 else len(css)), ( + "the mobile block squeezes the anchor like the path" + ) + assert "url(http" not in css and "@import url(" not in css, ( + "no CDN (AGENTS.md rule 6)" + ) + + # --------------------------------------------------------------------------- # Viewer URL builder — encoded query (spaces/slashes in real paths) # --------------------------------------------------------------------------- diff --git a/tests/unit/test_git_sources.py b/tests/unit/test_git_sources.py index 6188c6a..0818a74 100644 --- a/tests/unit/test_git_sources.py +++ b/tests/unit/test_git_sources.py @@ -179,3 +179,124 @@ def test_alias_env_fallback_unchanged(monkeypatch: pytest.MonkeyPatch) -> None: assert urls == ["https://env.example/one.git"] assert origin == "env" + + +# --- phase 145 (task 01): origin_for_source (the read result's origin) --- + + +class _GitOnlySession: + """The origin resolver's session (phase 145, task 01): emulates the + resolver's ``kind = 'git'`` WHERE clause — only the git rows ever + reach the caller, exactly like the real query; the seeded local + rows prove they are never candidates.""" + + def __init__(self, rows: list[GitSource]) -> None: + self._rows = [row for row in rows if row.kind == "git"] + self.statements: list[Any] = [] + + def scalars(self, stmt: Any) -> _FakeScalars: + self.statements.append(stmt) + return _FakeScalars(self._rows) + + +# --- the basename mapping -------------------------------------------------- + + +def test_origin_basename_mapping() -> None: + """The URL's path component — after the ``://``, or after the ``:`` + for scp-style ``git@host:owner/repo`` — trailing ``/`` stripped, + a single trailing ``.git`` suffix stripped, last segment taken. An + ``@`` inside the PATH is not userinfo and stays (no split, no + strip).""" + cases = { + "https://github.com/owner/myrepo.git": "myrepo", # the .git suffix + "https://github.com/owner/myrepo": "myrepo", # no suffix, same dir + "https://github.com/owner/myrepo/": "myrepo", # trailing slash + "https://github.com/owner/myrepo.git/": "myrepo", # both + "git@github.com:owner/myrepo.git": "myrepo", # scp-style + suffix + "git@github.com:owner/myrepo": "myrepo", # scp-style, no suffix + "ssh://git@host/owner/myrepo.git": "myrepo", # ssh scheme, @ in the host part + "https://host/a@b/myrepo.git": "myrepo", # @ inside the PATH — not userinfo + } + for url, expected in cases.items(): + assert resolver._url_basename(url) == expected, url + + +# --- the match: sanitize + branch ------------------------------------------ + + +def test_origin_exact_match_is_sanitized() -> None: + """EXACTLY ONE match → the phase-121 masked URL + the branch — a + legacy embedded-credential row never leaks its secret (the stored + value is untouched; the OUTPUT mask does the work).""" + session = _GitOnlySession( + [GitSource(url="https://user:tok@host/x.git", kind="git", branch=None)] + ) + + assert resolver.origin_for_source(session, "x") == ("https://host/x.git", None) # pyright: ignore[reportArgumentType] + + +def test_origin_branch_passthrough() -> None: + """The branch rides through as-is: pinned → the name (the read + result's ``branch:`` line), NULL → ``None`` (the default branch — + no ``branch:`` line).""" + session = _GitOnlySession( + [GitSource(url="https://host/x.git", kind="git", branch="release/1.2")] + ) + + assert resolver.origin_for_source(session, "x") == ("https://host/x.git", "release/1.2") # pyright: ignore[reportArgumentType] + + +def test_origin_local_rows_never_origins() -> None: + """Local rows are never origins (nothing to link to from the user's + browser): a source that exists ONLY as a local row gets no origin + (zero git candidates); a local + git pair resolves to the git row.""" + local = GitSource(url="/srv/repo", kind="local", path="/srv/repo") + session = _GitOnlySession([local]) # the WHERE clause drops it + assert resolver.origin_for_source(session, "repo") is None # pyright: ignore[reportArgumentType] + + session = _GitOnlySession( + [local, GitSource(url="https://host/repo.git", kind="git", branch=None)] + ) + assert resolver.origin_for_source(session, "repo") == ("https://host/repo.git", None) # pyright: ignore[reportArgumentType] + + +# --- the no-guess rows: None + WARNING -------------------------------------- + + +def test_origin_zero_match_is_none_with_warning( +caplog: pytest.LogCaptureFixture, +) -> None: + """No git row's basename matches (a plain local source) → ``None`` + + a WARNING naming the source — the read result stays + byte-identical to pre-phase.""" + session = _GitOnlySession([GitSource(url="https://host/other.git", kind="git")]) + + assert resolver.origin_for_source(session, "repo") is None # pyright: ignore[reportArgumentType] + + warnings = [record for record in caplog.records if record.levelname == "WARNING"] + assert len(warnings) == 1 + assert "repo" in warnings[0].message + + +def test_origin_ambiguous_is_none_with_warning( +caplog: pytest.LogCaptureFixture, +) -> None: + """TWO-OR-MORE rows share the basename → ``None`` + a WARNING naming + the sanitized candidates (never guess — a wrong origin would ride + into every answer about the document). The candidates are logged + SANITIZED: a legacy embedded credential stays out of the log too.""" + session = _GitOnlySession( + [ + GitSource(url="https://user:tok@a.example/org/repo.git", kind="git"), + GitSource(url="git@b.example:other/repo.git", kind="git"), + ] + ) + + assert resolver.origin_for_source(session, "repo") is None # pyright: ignore[reportArgumentType] + + warnings = [record for record in caplog.records if record.levelname == "WARNING"] + assert len(warnings) == 1 + assert "https://a.example/org/repo.git" in warnings[0].message + assert "git@b.example:other/repo.git" in warnings[0].message + assert "tok" not in warnings[0].message diff --git a/tests/unit/test_prompt_lock.py b/tests/unit/test_prompt_lock.py index 6baac77..0fe6186 100644 --- a/tests/unit/test_prompt_lock.py +++ b/tests/unit/test_prompt_lock.py @@ -15,6 +15,7 @@ from __future__ import annotations import hashlib from app.rag.prompts import ( + ORIGIN_RULE, PERSONA, SUGGEST_INTRO, TOOLS_SECTION, @@ -77,12 +78,18 @@ def test_high_and_low_bases_byte_locked() -> None: #: form — do not re-read a document read IN FULL (answer from it); #: a PARTIAL document (a range read, or a truncated read) is not #: complete in the context until read all — read the other parts with -#: ``offset``/``limit``. Only that sentence moved — the prefix (the -#: ``ls``-clause opening) and the suffix (the discipline-rules -#: ending) survived byte-identical, so they are the same anchors as -#: pre-phase-141. -TOOLS_SECTION_SHA256 = "3671680bbf4e04efe1186690c18d24869e4974ceb29485cedc3e34a9286bf3b2" -TOOLS_SECTION_LEN = 2656 +#: ``offset``/``limit``. RE-CUT again for phase 145 (task 02): the +#: ``read`` clause gains the original-source rule — :data:`ORIGIN_RULE`, +#: the one sentence-group appended at the ``read`` clause's end (the +#: truncation workflow sentence) and BEFORE the ``grep`` clause begins +#: (the ``origin:`` line is a read-result fact; the rule binds the +#: ANSWER that uses what the read added). Only that sentence-group was +#: inserted — the prefix (the ``ls``-clause opening) and the suffix +#: (the discipline-rules ending) survived byte-identical, so they are +#: the same anchors as pre-phase-145; the pre-phase-145 sha/length are +#: kept below for the diff-style pin. +TOOLS_SECTION_SHA256 = "41a84885be98d4e15b95205694d59d364296a30e27758d39ae737881d5bacaa7" +TOOLS_SECTION_LEN = 3279 TOOLS_SECTION_PREFIX = ( "\n" "You may extend your context with three tools. `ls` lists the " @@ -94,6 +101,33 @@ TOOLS_SECTION_SUFFIX = ( "you need.\n" ) +#: The pre-phase-145 anchors — the section BEFORE :data:`ORIGIN_RULE` was +#: inserted. Kept for the diff-style pin: the ONLY byte change is the +#: rule's insertion, so removing the rule reproduces these exact bytes. +TOOLS_SECTION_PRE_PHASE_145_SHA256 = ( + "3671680bbf4e04efe1186690c18d24869e4974ceb29485cedc3e34a9286bf3b2" +) +TOOLS_SECTION_PRE_PHASE_145_LEN = 2656 + + +#: Phase 145 (task 02) — the original-source rule the ``read`` teaching +#: carries is :data:`app.rag.prompts.ORIGIN_RULE` (imported above, LOCKED +#: verbatim). :data:`TOOLS_SECTION` embeds that exact sentence-group at +#: the ``read`` clause's end (the truncation workflow sentence) and +#: BEFORE the ``grep`` clause begins. A read result for a git-sourced +#: document carries an ``origin:`` line (and a ``branch:`` line when a +#: branch is pinned) — the read contract's fact — so when the answer uses +#: such a document it includes the original source in the answer: a link +#: to the repository (the ``origin`` line) with the document's path (the +#: ``Document …`` header) named and the pinned branch when one is shown, +#: as the repository's file-view URL where the model can tell it +#: (``/blob//``), or the repository URL plus the +#: path as plain text otherwise. The rule is CONDITIONAL on the ``origin`` +#: line's presence — a local-sourced document's read result carries none, +#: so the rule adds nothing to those answers (the model has no URL to +#: cite; it must not invent one — the sentence's conditioning does that +#: job). + def test_tools_section_byte_locked() -> None: """The ```` teaching is LOCKED verbatim too (the E2E mock keys @@ -105,6 +139,41 @@ def test_tools_section_byte_locked() -> None: assert TOOLS_SECTION.endswith(TOOLS_SECTION_SUFFIX) +def test_tools_section_origin_rule_verbatim_at_its_position() -> None: + """Phase 145 (task 02): the original-source rule is LOCKED verbatim + and sits at its EXACT position — the ``read`` clause's end (the + truncation workflow sentence) and BEFORE the ``grep`` clause begins. + The sentence-group is pinned byte-for-byte, and the surrounding + boundary (the truncation sentence's tail → the rule → the ``grep`` + clause's opening) is pinned as one contiguous substring, so a move + or re-spelling of ANY byte (the rule's or the boundary's) fails.""" + # Verbatim presence. + assert ORIGIN_RULE in TOOLS_SECTION + assert TOOLS_SECTION.count(ORIGIN_RULE) == 1 # exactly once + # Exact position: after the truncation sentence, before the grep + # clause — one contiguous span, no drift at either seam. + assert ( + "it searches the whole document. " + + ORIGIN_RULE + + " `grep` locates an exact string" + ) in TOOLS_SECTION + + +def test_tools_section_origin_rule_is_the_only_byte_change() -> None: + """Phase 145 (task 02) diff-style pin: the section's OTHER bytes are + byte-identical to pre-phase — the pre-phase section with ONLY the new + sentence-group inserted. Removing the rule (and the single space + that now separates it from the ``grep`` clause) reproduces the + pre-phase-145 section byte-for-byte (sha + length), and the length + delta is exactly the rule's length plus that one inserted space.""" + assert len(TOOLS_SECTION) == ( + TOOLS_SECTION_PRE_PHASE_145_LEN + len(ORIGIN_RULE) + 1 + ) + pre_phase = TOOLS_SECTION.replace(ORIGIN_RULE + " ", "") + assert len(pre_phase) == TOOLS_SECTION_PRE_PHASE_145_LEN + assert _sha256(pre_phase) == TOOLS_SECTION_PRE_PHASE_145_SHA256 + + # ---------- the LOW prompt's locked DEFLECT_MODE body ---------- #: The ``DEFLECT_MODE`` body exactly as it was pre-phase-112 (the diff --git a/tests/unit/test_prompts.py b/tests/unit/test_prompts.py index b372b54..0a2f2f2 100644 --- a/tests/unit/test_prompts.py +++ b/tests/unit/test_prompts.py @@ -48,6 +48,7 @@ import pytest from app.config import Settings from app.models import Document from app.rag.prompts import ( + ORIGIN_RULE, PERSONA, SHOW_IMAGE_CLAUSE, SUGGEST_INTRO, @@ -463,6 +464,43 @@ def test_tools_section_phase72_clauses_in_high_prompt_not_low() -> None: assert "including the source name" not in low +def test_tools_section_phase145_origin_rule() -> None: + """Phase 145 (task 02): the ```` ``read`` clause carries the + original-source rule (LOCKED verbatim in :data:`ORIGIN_RULE`) — when + the answer uses a document whose read result carries an ``origin:`` + line, it includes the original source (a link to the repository + from the origin line, with the document's path and the pinned branch + where shown) so the user opens the complete original in one step. + The rule rides the HIGH prompt (```` is HIGH-only) and never + the LOW/deflection prompt. The citation discipline is UNCHANGED — + the rule extends the ANSWERS of cited documents; it does not widen + the cite set (the "cite only what you read" sentence stays + byte-identical — pinned in ``test_prompt_lock``).""" + # The rule is present verbatim in the constant, exactly once, and at + # its exact position (the read clause's end, before the grep clause). + assert TOOLS_SECTION.count(ORIGIN_RULE) == 1 + assert ( + "it searches the whole document. " + + ORIGIN_RULE + + " `grep` locates an exact string" + ) in TOOLS_SECTION + # It rides the built HIGH prompt with the rest of the section. + doc = _doc("kubernetes.md", "Talos Linux on three nodes.", "Kubernetes") + high = build_high_prompt([doc]) + assert ORIGIN_RULE in high + # ...and never the LOW/deflection prompt (```` is HIGH-only). + for low in (build_deflect_prompt(["T1"]), build_deflect_prompt([])): + assert ORIGIN_RULE not in low + # The citation discipline is UNCHANGED: the rule does not widen the + # cite set (the "cite only what you read" sentence is untouched). + assert ( + "Cite only the document(s) you read — or, if you answered from a " + "suggested summary without reading it, cite that suggested " + "document — never a document you neither read nor used." + ) in high + assert "Cite only the document(s) you read" in SUGGEST_INTRO + + def test_documents_section_leads_with_the_suggest_intro() -> None: """Phase 118, task 03: the ```` section leads with the start-here :data:`SUGGEST_INTRO` line BEFORE the first block (the