phase: 145_git_source_links
Build and Push Containers / build-and-push-app (push) Successful in 2m23s
Build and Push Containers / build-and-push-db (push) Successful in 13s

**Phase 145 final verification pass — all green, no code changes needed**

- **Verified (no defects found):** resolver `origin_for_source` + `_url_basename` (scp/`.git`/trailing-slash, sanitize, ambiguity WARNING); `origin:`/`branch:` lines in all four read shapes via shared `_origin_lines`; HIGH `<tools>` origin rule (verbatim + only-byte-change pins); `DocContent.origin_url/origin_branch` (absent-never-null serializer); `renderDocument` "Original" anchor (copyable verbatim URL, `_blank`/`noopener`, branch in title); mock scripted origin-link flow (disjoint trigger — grep-verified vs all other E2E files)
- **Tests:** `uv run pytest --cov=app --cov-report=term-missing` → 3402 passed, `app/` coverage **99%** (>90% gate); `uv run pytest tests/e2e/test_git_source_links.py -v --no-cov` → **4 passed** in isolation; `uv run ruff check .` → clean; `uv run pyright` → 0 errors, 0 warnings
- **Completion criteria:** (1) git read results carry origin/branch, local byte-identical — ✅ (unit pins + E2E control); (2) HIGH `<tools>` rule pinned, LOW byte-identical — ✅; (3) modal AND /document.html show Original link, branch in label when pinned, nothing for local — ✅; (4) full suite green + coverage + E2E + lint/types — ✅; (5) no completed-phase behavior change (read header line 1, mock prefix, done frame/SourceRef untouched, LOW sha) — ✅
- **Extra regression check:** re-ran shared-mock E2E suites `test_read_line_ranges.py` (2 passed) and `test_document_viewer.py` (8 passed) — no cross-suite regressions from the `mock_llm.py` addition
- **Documentation:** README.md reviewed in full (env table ⊇ `.env.example`, pages/routes, config defaults, scripts, 17-color palette, 300 s guard, 2 s sync poll, phase-145 provenance section matches implementation) and AGENTS.md (72rem container, no-CDN test, smoke test, gitignore, gates) — **verified consistent, no changes needed**
- **Notable:** phase-116 screenshot PNG was regenerated during this phase's E2E runs (byte re-encode only, visually identical, correct no-anchor local-source meta row) — left in tree for the harness commit
- **Next pending phase:** `146_search_tool_scope` (then 147, 148, 149)
This commit is contained in:
2026-10-01 11:13:43 -04:00
parent 486ceb4019
commit 32a3ec4f5e
37 changed files with 2856 additions and 22 deletions
+242
View File
@@ -714,6 +714,55 @@ Implements just enough of the aipi surface:
deflected turns never hit it); verified 2026-10-01: no existing
E2E question or fixture file contains the phrase, so every
other suite is unaffected.
- user message containing ``original source link``
(``ORIGIN_LINK_TRIGGER``, phase 145 task 04 — the git-source-
links' dedicated story suite
``tests/e2e/test_git_source_links.py``) **and** the system
prompt carries the ``<tools>`` section -> the deterministic
SCRIPTED ORIGIN-LINK flow (the owner's scenario, TODO.md L7:
the answer grounded in a git-sourced document must carry the
way to the complete original in ONE step): the question
carries its own tool call after the colon — ``original source
link: read source/path`` — the question-carried-call
convention (``_READ_CAP_CALL_RE``), then discriminated
statelessly from the tool results (streaming only):
* request 1 (``tools`` offered, no tool results yet): a
``tool_calls`` delta — ``read`` on the parsed target
(synthetic id ``call_0``);
* a ``tool``-role READ result is in the messages (content
starting with the agent's ``"Document "`` prefix —
``_READ_RESULT_PREFIX``; the phase-145 ``origin:`` /
``branch:`` lines sit BELOW the header, so the first-line
parsing is untouched): the deterministic FINAL answer —
the repository URL VERBATIM, parsed from the read
result's ``origin:`` line BY THE MOCK ITSELF (the mock
cannot fabricate it — the real server's resolver put the
line in the result, so the URL in the answer is the
server-side half's assertion): ``The full source lives at
<url> — see the repo for the complete original.``
(``ORIGIN_LINK_ANSWER``) followed by the house
scripted-turn echo — the read result VERBATIM under
``Here's what the read returned:`` (the rendered output
is the E2E's lens on the ``origin:``/``branch:`` lines'
presence);
* a tool result is in the messages that carries NO
``origin:`` line (a local-sourced document — the byte-
identity control): the deterministic NO-ORIGIN answer —
``No origin line in the read result — the document is not
from a git source.`` (``ORIGIN_LINK_NO_ORIGIN``) + the
same verbatim echo (the suite asserts the line's ABSENCE
through it);
* a tool result is in the messages that is NOT a read
result (a refusal — the premise broke): the
deterministic broken-premise echo — the offending result
VERBATIM under ``Premise broke:`` (the two-ranges
convention — the suite fails loudly on it).
Checked BEFORE the plain ``TOOLS_TRIGGER`` flow (disjoint
trigger phrases — the phase-71/72/94/118/131/139/141 ordering
convention; the trigger needs the ``<tools>`` section, so
deflected turns never hit it); verified 2026-10-01: no
existing E2E question or fixture file contains the phrase, so
every other suite is unaffected.
Failure injection (phase 67, LLM retry, TODO.md L3) — deterministic
dead-endpoint behavior for the retry E2E suite (``tests/e2e/
@@ -2667,6 +2716,148 @@ def _two_ranges_flow(body: dict[str, Any]) -> tuple[str, ...] | None:
return ("read1", target)
# ---------------------------------------------------------------------------
# Phase 145 (task 04): the git-source-links' dedicated story suite — the
# owner's "the answer points at the original git source" scenario. See
# the module docstring.
# ---------------------------------------------------------------------------
#: A user message containing this substring (case-insensitive) —
#: combined with the ``<tools>`` section in the system prompt —
#: drives the deterministic SCRIPTED ORIGIN-LINK flow (the
#: git-source-links' story suite,
#: ``tests/e2e/test_git_source_links.py`` — the owner's scenario,
#: TODO.md L7: the answer grounded in a git-sourced document must
#: carry the way to the complete original in ONE step). Checked
#: BEFORE the plain ``TOOLS_TRIGGER`` flow (disjoint trigger
#: phrases — the phase-71/72/94/118/131/139/141 ordering
#: convention; the trigger needs the ``<tools>`` section, so
#: deflected turns never hit it); verified 2026-10-01: no
#: existing E2E question or fixture file contains the phrase, so
#: every other suite is unaffected.
ORIGIN_LINK_TRIGGER = "original source link"
#: The scripted call in the origin-link question (case-insensitive
#: — the suite's questions capitalize the trigger's first
#: letter): the verb (``read``) plus the target — a combined
#: ``source/path``, parsed from the RAW user message so the target
#: keeps its case. The target is a ``[a-z0-9_./-]`` run (case-
#: insensitively), so the suite's `` — `` flavor separator (em
#: dash) can never bleed into it (the read-cap / drill-down
#: convention, ``_READ_CAP_CALL_RE`` / ``_DRILL_CALL_RE``).
_ORIGIN_LINK_CALL_RE = re.compile(
r"original source link:\s*read\s+(?P<arg>[a-z0-9_./-]+)",
re.I,
)
#: The flow's FINAL answer for a read result that carries the
#: phase-145 ``origin:`` line — the DETERMINISTIC string carrying
#: the repository URL VERBATIM (the model-side contract, owner-
#: confirmed 2026-09-30: the answer includes the original source).
#: ``{url}`` is parsed from the read result's ``origin:`` line BY
#: THE MOCK ITSELF — the mock cannot fabricate it, so the URL in
#: the answer is the server-side half's assertion (the real
#: server's resolver put the line in the result the model saw).
ORIGIN_LINK_ANSWER = (
"The full source lives at {url} — see the repo for the complete original."
)
#: The flow's FINAL answer for a read result that carries NO
#: ``origin:`` line (a local-sourced document — the byte-identity
#: control): the deterministic no-origin sentence (its wording
#: carries NO ``origin:`` substring — the suite's absence
#: assertion keys off the line's exact prefix).
ORIGIN_LINK_NO_ORIGIN = (
"No origin line in the read result — the document is not from a git source."
)
#: The house scripted-turn echo header (the ``READ_CAP_TRIGGER``
#: flow's convention): the read result rides VERBATIM into the
#: answer, so the rendered output is the E2E's lens on the
#: ``origin:``/``branch:`` lines' presence (and their absence in
#: the local-source control).
_ORIGIN_LINK_ECHO_HEADER = "Here's what the read returned:"
def _origin_link_url(result: str) -> str | None:
"""The read result's ``origin:`` line's URL (the phase-145
task-01 line — the sanitized repository URL the real server's
resolver put in the result) — or ``None`` when the result
carries no origin line (a local-sourced document, no git row,
or an ambiguous basename).
The search is position-independent (any line with the exact
``origin: `` prefix): the phase-145 lines sit between the
``date:`` line and the content, but the parse never drifts if
the result's header shape changes. A document BODY line that
merely starts with ``origin: `` would be a fixture defect (the
suite's fixtures are pinned content — no such line), never a
mock defect.
"""
for line in result.splitlines():
if line.startswith("origin: "):
return line[len("origin: "):]
return None
def _origin_link_flow(body: dict[str, Any]) -> tuple[str, ...] | None:
"""Classify a phase-145 scripted origin-link request (see the
module docstring). The question carries the scripted call
(``original source link: read source/path``); the step is then
discriminated statelessly from the tool results, like the
other marker flows:
* ``("call", target, "call_0")`` — ``tools`` are offered and
no ``tool``-role result is in the messages yet: the scripted
``read`` on the parsed target (synthetic id ``call_0``).
* ``("answer", url, result)`` — a ``tool``-role READ result
(content starting with the agent's ``"Document "`` prefix —
``_READ_RESULT_PREFIX``; the phase-145 ``origin:`` /
``branch:`` lines sit BELOW the header, so the first-line
parsing is untouched) is in the messages: the deterministic
FINAL answer — ``url`` the result's ``origin:`` line's URL
(or ``""`` — the no-origin sentence, the local-source
control; the empty-string sentinel keeps the return
homogeneous ``tuple[str, …]``, the ``_read_lines_flow``
convention) and ``result`` the read result VERBATIM (the
house scripted-turn echo — the rendered output is the lens
on the origin lines).
* ``("bad", result)`` — a tool result is in the messages that
is NOT a read result (a refusal — the premise broke): the
deterministic broken-premise echo — the offending result
VERBATIM under ``Premise broke:`` (the two-ranges convention
— the suite fails loudly on it).
* ``None`` — not the flow: the trigger is absent, the
``<tools>`` section is missing (deflected turns never carry
it), the scripted call is unparseable, or ``tools`` are not
offered and no tool results are in the messages yet (e.g.
``agent_max_rounds=0``).
"""
user = _user(body)
if ORIGIN_LINK_TRIGGER not in user.lower():
return None
if "<tools>" not in _system(body):
return None
match = _ORIGIN_LINK_CALL_RE.search(user)
if match is None:
return None
target = match.group("arg")
results = _tool_results(body)
if results:
last = results[-1]
if not last.startswith(_READ_RESULT_PREFIX):
return ("bad", last)
# The empty-string sentinel (never a real URL — a URL is
# always non-empty: the resolver returns the stored URL's
# sanitized form, which is a non-empty string): keeps the
# return homogeneous ``tuple[str, …]`` (pyright), the
# ``_read_lines_flow`` convention.
return ("answer", _origin_link_url(last) or "", last)
if not body.get("tools"):
return None
return ("call", target, "call_0")
def long_answer() -> str:
"""~900-word deterministic walkthrough (phase 11): numbered steps plus
a unique final line that must survive the stream untruncated."""
@@ -4044,6 +4235,57 @@ def chat_completions(body: dict[str, Any]) -> Any:
media_type="text/event-stream",
headers={"Cache-Control": "no-cache", "X-Accel-Buffering": "no"},
)
# Phase 145 (task 04): the owner's "the answer points at the
# original git source" scenario (the git-source-links' story
# suite, tests/e2e/test_git_source_links.py — TODO.md L7):
# the scripted read, then the final answer that carries the
# repository URL parsed from the read result's ``origin:``
# line (the server-side half — the mock cannot fabricate it)
# + the result VERBATIM (the lens on the origin lines'
# presence/absence). Checked BEFORE the plain TOOLS_TRIGGER
# flow (disjoint trigger phrases — the phase-71/72/94/118/
# 131/139/141 ordering convention; the trigger needs the
# ``<tools>`` section, so deflected turns never hit it).
origin_link = _origin_link_flow(body)
if origin_link is not None:
if origin_link[0] == "call":
stream = _tool_call_stream(
"read", {"path": origin_link[1]}, origin_link[2]
)
elif origin_link[0] == "bad":
# The premise broke (a refusal — the scripted read
# was refused): echo the offending result verbatim
# so a suite fails loudly on it (the two-ranges
# convention).
stream = _sse_stream(
_apply_max_tokens(
f"Premise broke:\n{origin_link[1]}",
body.get("max_tokens"),
),
0.0,
)
else: # "answer" — the final answer (tests 1–3's model-
# side contract) + the verbatim echo (the lens on
# the origin lines' presence/absence). ``url`` is
# the ``origin:`` line's URL or the "" sentinel
# (the no-origin sentence — the local-source
# control).
url = origin_link[1]
sentence = (
ORIGIN_LINK_ANSWER.format(url=url) if url else ORIGIN_LINK_NO_ORIGIN
)
stream = _sse_stream(
_apply_max_tokens(
f"{sentence}\n{_ORIGIN_LINK_ECHO_HEADER}\n{origin_link[2]}",
body.get("max_tokens"),
),
0.0,
)
return StreamingResponse(
stream,
media_type="text/event-stream",
headers={"Cache-Control": "no-cache", "X-Accel-Buffering": "no"},
)
# Phase 124 (task 07, image chips and answers, TODO L3): the
# scripted SHOW_IMAGE flow — the mock shows the seeded image
# document (the ``<documents>`` block with the image-
+585
View File
@@ -0,0 +1,585 @@
"""Phase 145 task 04 E2E (Playwright, mock-only): the git-source-links'
dedicated story suite — the owner's scenario, end to end.
The browser-visible proof (TODO.md L7: "there always needs to be a
link to the original git source if possible … point to the skills
repo in the answer so I can go download the skills"): a question
about a git-sourced document produces (1) an answer that contains
the repository link (the model-side teaching, through the real read
result that carries the ``origin:`` line) and (2) a document modal
whose meta row carries the "Original" external anchor. A
local-sourced document produces neither (the byte-identity guard).
Run in isolation (DB must be up: ``podman compose up -d db``):
uv run pytest tests/e2e/test_git_source_links.py -v --no-cov
MOCK-ONLY suite: ``E2E_REAL_LLM=1`` is not supported — the real
``turbo`` does whatever it does with the tools, while this story's
gate is the deterministic SCRIPTED flow in ``tests/e2e/mock_llm.py``
(the mock is the only E2E lens on the LLM's context):
* ``ORIGIN_LINK_TRIGGER`` — a user message containing ``original
source link`` **and** the system prompt carrying the HIGH
prompt's ``<tools>`` section. The question carries its own
scripted call after the colon — ``original source link: read
source/path`` (the question-carried-call convention,
``_READ_CAP_CALL_RE``). Request 1 is the scripted ``read``;
request 2 (the read result present) is the deterministic FINAL
answer — the repository URL VERBATIM, parsed from the read
result's ``origin:`` line BY THE MOCK (it cannot fabricate it —
the real server's resolver put the line in the result, so the
URL in the answer is the server-side half's assertion) —
followed by the house scripted-turn echo (the read result
VERBATIM — the rendered output is the lens on the
``origin:``/``branch:`` lines' presence; the local-source
control's read result carries none, and the mock answers with
its deterministic no-origin sentence instead).
KB fixture — TWO local directories (the house per-module import —
the ``test_document_viewer.py`` importer-in-a-thread pattern), each
holding ONE readable markdown document:
* ``skills/phased-execution.md`` — the owner's scenario document
(the phased execution process for pi.dev). The source's name is
``skills`` — it MATCHES the seeded git row's URL basename
(``https://example.com/reese/skills`` → ``skills``: the path
component, trailing ``/`` stripped, no ``.git`` suffix to strip,
last segment — the resolver's name mapping). The fixture document
is locally imported, but the mapping is name-based — which is the
point of the seam;
* ``control/notes.md`` — a SECOND plain local source with NO git
row: the byte-identity control (its read result and its modal
meta row must be exactly the pre-phase shapes).
The git row is seeded in the suite's OWN DB session (a direct
``SessionLocal`` write, per test — ``branch=None`` for tests 1–2,
``branch="main"`` for the pinned-branch test, and ABSENT for the
control test) — never through the API, and never during the import
(a ``kind='git'`` row present at import time would be cloned —
``example.com`` is not a git host; the resolver reads the table
LIVE at read/content time, so seeding after the import is the
whole trick).
The app under test is the conftest session app (mock LLM + the
mock-calibrated gate: threshold 0.30 + lexical floor 0.15): both
questions share the subject words with their own fixture document
(the mock's token-overlap distribution), so every scripted turn
stays GROUNDED (the ``<tools>`` section is present) and the read
document is the turn's cited chip (phase 119 A1 — the done frame's
``sources`` is the READ docs).
Test → observable mapping (Playwright Mapping Rule; the story is
the owner's TODO item — one Playwright file per story, A16):
1. ``test_answer_carries_origin_link`` — the ``original source
link`` question on the git-matched source's document → the
rendered tool output for the ``read`` (the mock's verbatim echo
in the answer — the house scripted-turn lens) carries
``origin: https://example.com/reese/skills`` (the server put
the line in the result the model saw) and the final ``done``
answer contains the repository URL (the scripted echo — the
model-side contract, deterministic).
2. ``test_modal_shows_original_anchor`` — same setup → click the
source chip → ``#doc-modal`` rendered → the meta row contains
the ``.doc-origin`` anchor with ``href`` = the URL,
``target="_blank"``, ``rel="noopener"``, ``textContent`` = the
URL verbatim (and NO branch in the text — ``branch=None``: the
``title`` is the bare URL).
3. ``test_pinned_branch_in_title`` — the ``branch="main"`` row →
the read result's surface also carries ``branch: main``, the
done answer still carries the repository URL, and the same
anchor's ``title`` is
``https://example.com/reese/skills — branch: main`` (the text
stays the bare URL — the copyable-link contract).
4. ``test_local_source_has_no_origin`` — the control source's
document (no git rows at all — the pre-phase world) → the
read's rendered tool output carries NO ``origin:`` line (the
byte-identical result shape — the mock's no-origin sentence
answers) and the modal's meta row has NO ``.doc-origin``
anchor (the pre-phase badge row).
"""
from __future__ import annotations
import asyncio
import json
import time
from collections.abc import Iterator
from pathlib import Path
from threading import Thread
from typing import Any
import pytest
from playwright.sync_api import Page, expect
from sqlalchemy import delete, func, select, text
from app.config import Settings
from app.db import SessionLocal
from app.models import Document, GitSource
from app.rag.importer import import_sources
from app.rag.llm import LLMClient
from e2e.auth_helpers import login
from e2e.mock_llm import ORIGIN_LINK_ANSWER, ORIGIN_LINK_NO_ORIGIN
REPO = Path(__file__).resolve().parents[2]
SOURCE = "skills" # the local directory's basename = the source name
SKILLS_REL = "phased-execution.md"
SKILLS_SP = f"{SOURCE}/{SKILLS_REL}" # the combined read identity
CONTROL = "control" # the plain local control source (NO git row)
CONTROL_REL = "notes.md"
CONTROL_SP = f"{CONTROL}/{CONTROL_REL}"
#: The seeded git row's public URL (no credential — the public form;
#: the phase-121 sanitizer is a byte-identical no-op for it). Its
#: URL basename — ``skills`` — MATCHES the fixture source's name
#: (the resolver's name mapping — the seam).
ORIGIN_URL = "https://example.com/reese/skills"
PINNED_BRANCH = "main"
SKILLS_DOC = """# Phased execution for pi.dev
How does the phased execution process work for pi.dev? It runs one
task at a time: each task file executes in a separate pi subprocess,
and the harness manages phase and task files, retries, and final
validation.
## Task files
Every task file is a unit of work. The pipeline runs the todo phases
in numeric order, and a completed phase moves to the complete
directory with its own test suite green.
## Validation
A phase is complete only when the unit and integration tests pass,
the coverage gate is green, and the Playwright story suite passes
in isolation. The harness then moves the task file and commits the
phase atomically.
"""
CONTROL_DOC = """# Control notes
This is a plain local directory source. It is walked directly by
the importer and never cloned from a repository. The control note
exists so the byte-identity guard has a document: local documents
read, render, and cite exactly as before, with nothing new in the
read result or the meta row.
"""
# The suite's own content pins — the mock's echo assertion keys off
# the read result's ``origin:``/``branch:`` lines, so neither
# fixture may contain a line with either prefix (a fixture defect,
# not a mock defect — see the mock's ``_origin_link_url``).
assert "origin:" not in SKILLS_DOC and "branch:" not in SKILLS_DOC
assert "origin:" not in CONTROL_DOC and "branch:" not in CONTROL_DOC
# The scripted questions — the mock's ``ORIGIN_LINK_TRIGGER`` + the
# question-carried call (``_ORIGIN_LINK_CALL_RE``) + the subject
# words (the mock-calibrated gate: both questions stay GROUNDED on
# their own fixture document).
Q_SKILLS = (
f"Original source link: read {SKILLS_SP} — "
"how does the phased execution process work for pi.dev?"
)
Q_CONTROL = (
f"Original source link: read {CONTROL_SP} — "
"what does the control note say about plain local sources?"
)
#: The final answer the mock scripts for the git-matched document
#: (the ``ORIGIN_LINK_ANSWER`` constant with the seeded URL — the
#: model-side contract, imported so the mock can never drift).
ANSWER_SENTENCE = ORIGIN_LINK_ANSWER.format(url=ORIGIN_URL)
# --------------------------------------------------------------------------
# Fixtures
# --------------------------------------------------------------------------
@pytest.fixture(scope="module")
def kb_dirs(tmp_path_factory: pytest.TempPathFactory) -> tuple[Path, Path]:
"""The story's two local-directory sources (host temp dirs — the
app server runs on the same host, so the paths are visible to
it): the ``skills`` source (its basename MATCHES the seeded git
row's URL basename — the resolver's name mapping) and the
``control`` source (no git row — the byte-identity control)."""
skills = tmp_path_factory.mktemp("bor_origin_skills") / SOURCE
skills.mkdir()
(skills / SKILLS_REL).write_text(SKILLS_DOC, encoding="utf-8")
control = tmp_path_factory.mktemp("bor_origin_control") / CONTROL
control.mkdir()
(control / CONTROL_REL).write_text(CONTROL_DOC, encoding="utf-8")
return skills, control
def _run_in_thread(coro: Any) -> Any:
"""Run a coroutine on a worker thread (the
``test_document_viewer`` pattern — Playwright's sync API keeps
an asyncio loop on the test thread, so ``asyncio.run`` cannot
run directly in a test body)."""
box: dict[str, Any] = {}
def runner() -> None:
try:
box["value"] = asyncio.run(coro)
except BaseException as e: # noqa: BLE001 — re-raised in the test
box["error"] = e
t = Thread(target=runner)
t.start()
t.join()
if "error" in box:
raise box["error"]
return box["value"]
async def _import_sources(roots: list[Path], mock_port: int) -> None:
"""The house per-module import (the ``test_document_viewer``
pattern): the REAL importer against the fixture roots under the
deterministic mock embeddings — the mock LLM supplies the
embeds, and the owner's local ``.env`` is NOT read (the
``_env_file=None`` isolation)."""
kwargs: dict[str, Any] = {
"_env_file": None,
"llm_base_url": f"http://127.0.0.1:{mock_port}/v1",
}
settings = Settings(**kwargs) # pyright: ignore[reportCallIssue]
await import_sources(roots, LLMClient(settings))
def _truncate_all() -> None:
"""Fresh registry + KB (the E2E isolation pattern): the E2E
suites share one Postgres, so a leftover git_sources row or
document would pollute the resolver and the retrieval the
scripted turns run against (the two-source KB's gate margins
are pinned against EXACTLY these two documents)."""
with SessionLocal() as db:
db.execute(
text(
"TRUNCATE chunks, documents, query_log, steering_notes, "
"kb_overview, git_sources, folder_summaries"
)
)
db.commit()
@pytest.fixture(scope="module")
def seeded_kb(mock_llm: int, kb_dirs: tuple[Path, Path]) -> None:
"""The story's precondition: the two-source KB (one readable
markdown doc each) imported under the deterministic mock, the
stored content byte-pinned. The git row is NOT part of the
import (a git row present at import time would be cloned — the
row is seeded in the suite's own DB session per test, AFTER
the import; the resolver reads the table live at
read/content time)."""
_truncate_all()
_run_in_thread(_import_sources(list(kb_dirs), mock_llm))
with SessionLocal() as db:
docs = {
(d.source, d.path): d for d in db.scalars(select(Document)).all()
}
assert set(docs) == {(SOURCE, SKILLS_REL), (CONTROL, CONTROL_REL)}, (
docs.keys()
)
assert docs[(SOURCE, SKILLS_REL)].content == SKILLS_DOC
assert docs[(CONTROL, CONTROL_REL)].content == CONTROL_DOC
# The import ran with NO git rows (the local-only KB): the
# seeded row lands after, per test.
assert db.scalar(select(func.count()).select_from(GitSource)) == 0
def _seed_git_row(branch: str | None, exist: bool = True) -> None:
"""The git row in the suite's OWN DB session (the task's seam):
delete every row, then — when *exist* — insert the one
``kind='git'`` row whose URL basename matches the ``skills``
fixture source (public URL, no credential; *branch* pinned or
NULL). The resolver reads the table LIVE — the row never
touches the importer."""
with SessionLocal() as db:
db.execute(delete(GitSource))
if exist:
db.add(GitSource(url=ORIGIN_URL, kind="git", branch=branch))
db.commit()
@pytest.fixture(autouse=True)
def _clean(db_ready: None) -> Iterator[None]:
"""Per-test query_log isolation (the KB itself is module-scoped
— the scripted turns never change it, so the documents persist
across the tests of this module)."""
with SessionLocal() as db:
db.execute(text("TRUNCATE query_log"))
db.commit()
yield
with SessionLocal() as db:
db.execute(text("TRUNCATE query_log"))
db.commit()
# --------------------------------------------------------------------------
# Page helpers (the test_read_line_ranges / test_document_viewer
# house patterns)
# --------------------------------------------------------------------------
#: Captures the raw SSE ``data:`` payloads of the /api/chat stream
#: (a response clone read in the background) — wire-level
#: assertions for the ``tool`` / ``done`` frames, independent of
#: the UI rendering.
SSE_HOOK = """
() => {
if (window.__sseInstalled) return;
window.__sseInstalled = true;
window.__sseFrames = [];
const origFetch = window.fetch;
window.fetch = async function (...args) {
const res = await origFetch.apply(this, args);
try {
const url = typeof args[0] === 'string' ? args[0] : args[0].url;
if (url.includes('/api/chat')) {
res.clone().text().then((bodyText) => {
for (const block of bodyText.split('\\n\\n')) {
const line = block.trim();
if (line.startsWith('data: ')) {
window.__sseFrames.push(line.slice(6));
}
}
});
}
} catch (e) { /* non-clonable responses: ignored */ }
return res;
};
}
"""
def _install_page_hooks(page: Page) -> None:
page.evaluate(SSE_HOOK)
def _frames(page: Page, deadline_s: float = 120.0) -> list[dict]:
"""The SSE frames captured since the last submit (``_submit``
clears the buffer), once the hook's background read settles."""
deadline = time.monotonic() + deadline_s
while True:
raw = page.evaluate("() => window.__sseFrames || []")
parsed = [json.loads(line) for line in raw if line]
if any(f.get("type") == "done" for f in parsed):
return parsed
if time.monotonic() > deadline:
raise AssertionError(
f"SSE hook captured no `done` frame (frames so far: "
f"{len(parsed)}) — hook install failed?"
)
time.sleep(0.05)
def _tool_frames(frames: list[dict]) -> list[dict]:
return [f for f in frames if f.get("type") == "tool"]
def _submit(page: Page, question: str) -> None:
page.evaluate("window.__sseFrames = []")
page.fill("#message-input", question)
page.click("#send-btn")
# The user bubble lands synchronously with the submit handler.
expect(page.locator(".msg.user .bubble").last).to_contain_text(question)
def _wait_settled(page: Page, timeout_ms: int = 120_000) -> None:
"""The turn is complete: answer text in the bubble, button
recovered (the phase-48 settle wait)."""
expect(page.locator(".msg.brain .bubble").last).not_to_have_text(
"", timeout=timeout_ms
)
expect(page.locator("#send-btn")).to_be_enabled(timeout=timeout_ms)
expect(page.locator("#send-label")).to_have_text("Send", timeout=timeout_ms)
def _bubble_text(page: Page) -> str:
"""The turn's rendered answer (the brain bubble's text content
— the mock's scripted echo lands here: the house scripted-turn
lens on the read result's ``origin:``/``branch:`` lines)."""
return page.locator(".msg.brain .bubble").last.text_content() or ""
def _open_modal_for_chip(page: Page, chip_text: str) -> None:
"""Drive the story's modal surface: the turn's cited chip
(phase 119 A1 — done.sources is the READ docs, so the read
document is the turn's ONLY chip) opens the same-page document
modal (the ``test_document_viewer`` contract)."""
chip = page.locator(".msg.brain .source-chip", has_text=chip_text)
expect(chip).to_have_count(1, timeout=30_000)
chip.first.click()
expect(page.locator("#doc-modal")).not_to_have_attribute("hidden")
expect(page.locator("#doc-modal")).to_be_visible()
# --------------------------------------------------------------------------
# 1. The answer carries the repository link (the model-side
# contract, through the real read result's ``origin:`` line)
# --------------------------------------------------------------------------
def test_answer_carries_origin_link(
page: Page, app_url: str, seeded_kb: None, db_ready: None
) -> None:
_seed_git_row(branch=None)
page.set_default_timeout(60_000)
login(page, app_url, next="/")
_install_page_hooks(page)
_submit(page, Q_SKILLS)
_wait_settled(page)
frames = _frames(page)
# The scripted read on the wire — the turn's only tool call.
assert _tool_frames(frames) == [
{"type": "tool", "name": "read", "argument": SKILLS_SP}
], _tool_frames(frames)
done = next(f for f in frames if f.get("type") == "done")
assert done["deflected"] is False, done
# The read document is the turn's cited source (the chip the
# modal tests click — phase 119 A1).
assert any(
s["source"] == SOURCE and s["path"] == SKILLS_REL for s in done["sources"]
), done["sources"]
# The read result's surface (the mock's verbatim echo in the
# answer — the house scripted-turn lens): the sanitized
# ``origin:`` line the SERVER put in the result the model saw
# (the task-01 resolver's output — the git row's URL,
# credential-free).
text = _bubble_text(page)
assert f"origin: {ORIGIN_URL}" in text, text
# The done answer carries the repository URL (the scripted
# echo — the model-side contract, deterministic): the mock
# parsed the URL from the origin line — it cannot appear in
# the answer unless the server sent it.
assert ANSWER_SENTENCE in text, text
# --------------------------------------------------------------------------
# 2. The modal's meta row shows the "Original" external anchor
# --------------------------------------------------------------------------
def test_modal_shows_original_anchor(
page: Page, app_url: str, seeded_kb: None, db_ready: None
) -> None:
_seed_git_row(branch=None)
page.set_default_timeout(60_000)
login(page, app_url, next="/")
_install_page_hooks(page)
_submit(page, Q_SKILLS)
_wait_settled(page)
_open_modal_for_chip(page, SKILLS_REL)
expect(page.locator("#doc-modal-title")).to_have_text(
"Phased execution for pi.dev"
)
# The meta row's "Original" anchor — the task-03 contract
# (ONE shared renderDocument site: the modal AND
# /document.html show it).
anchor = page.locator("#doc-modal-meta .doc-origin")
expect(anchor).to_have_count(1)
expect(anchor).to_have_attribute("href", ORIGIN_URL)
expect(anchor).to_have_attribute("target", "_blank")
expect(anchor).to_have_attribute("rel", "noopener")
# The copyable-link contract: the full URL VERBATIM as the
# visible text — NO branch in the text (branch=None), and the
# title is the bare URL (the pathCode precision idiom, no
# branch suffix).
expect(anchor).to_have_text(ORIGIN_URL)
expect(anchor).to_have_attribute("title", ORIGIN_URL)
# --------------------------------------------------------------------------
# 3. The pinned branch rides the read result and the anchor's title
# --------------------------------------------------------------------------
def test_pinned_branch_in_title(
page: Page, app_url: str, seeded_kb: None, db_ready: None
) -> None:
_seed_git_row(branch=PINNED_BRANCH)
page.set_default_timeout(60_000)
login(page, app_url, next="/")
_install_page_hooks(page)
_submit(page, Q_SKILLS)
_wait_settled(page)
# The read result's surface (the mock's verbatim echo): the
# origin line AND the pinned branch line (the task-01 second
# line, only when the row pins a branch).
text = _bubble_text(page)
assert f"origin: {ORIGIN_URL}" in text, text
assert f"branch: {PINNED_BRANCH}" in text, text
# The done answer still carries the repository URL (tests
# 1–3's model-side contract — the branch never joins the
# URL).
assert ANSWER_SENTENCE in text, text
_open_modal_for_chip(page, SKILLS_REL)
anchor = page.locator("#doc-modal-meta .doc-origin")
expect(anchor).to_have_count(1)
expect(anchor).to_have_attribute("href", ORIGIN_URL)
# The text stays the bare URL (copyable — the branch never
# joins the anchor text)…
expect(anchor).to_have_text(ORIGIN_URL)
# …the branch rides the title (the task-03 label contract).
expect(anchor).to_have_attribute(
"title", f"{ORIGIN_URL} — branch: {PINNED_BRANCH}"
)
# --------------------------------------------------------------------------
# 4. The local-source control is byte-identical (no ``origin:``
# line in the read result, no anchor in the meta row)
# --------------------------------------------------------------------------
def test_local_source_has_no_origin(
page: Page, app_url: str, seeded_kb: None, db_ready: None
) -> None:
_seed_git_row(branch=None, exist=False) # the pre-phase world: NO git rows at all
page.set_default_timeout(60_000)
login(page, app_url, next="/")
_install_page_hooks(page)
_submit(page, Q_CONTROL)
_wait_settled(page)
frames = _frames(page)
assert _tool_frames(frames) == [
{"type": "tool", "name": "read", "argument": CONTROL_SP}
], _tool_frames(frames)
done = next(f for f in frames if f.get("type") == "done")
assert done["deflected"] is False, done
# The byte-identity guard on the read result's surface (the
# mock's verbatim echo): the result carried NO origin line —
# the local source's result is byte-identical to pre-phase,
# so the mock's deterministic no-origin sentence answers.
text = _bubble_text(page)
assert ORIGIN_LINK_NO_ORIGIN in text, text
assert "origin:" not in text, text
assert "branch:" not in text, text
# …and the modal's meta row is the pre-phase badge row: NO
# .doc-origin anchor (the content payload's origin fields are
# absent on the wire — never null).
_open_modal_for_chip(page, CONTROL_REL)
expect(page.locator("#doc-modal-title")).to_have_text("Control notes")
expect(page.locator("#doc-modal-meta .doc-origin")).to_have_count(0)
# The pre-phase badges are untouched (the row still shows
# source · format · path · created · indexed · chunks).
expect(page.locator("#doc-modal-meta .doc-source-badge")).to_have_text(
CONTROL
)
expect(page.locator("#doc-modal-meta .doc-path")).to_have_text(
CONTROL_REL
)
expect(page.locator("#doc-modal-meta .doc-created")).not_to_be_empty()
+134 -2
View File
@@ -10,7 +10,13 @@ Uses the real compose Postgres (``db`` fixture) and FastAPI's TestClient:
tests and the anonymous 403/401 pins build their own client;
* GET: 404 for an unknown (source, path) pair;
* GET: 404 for traversal-style ``path`` values (no filesystem access → no
leak).
leak);
* GET: phase 145 (task 03) — the payload's git origin fields: a seeded
``kind='git'`` row (URL basename = the fixture source) adds
``origin_url`` (+ ``origin_branch`` for a pinned row); a legacy
embedded-credential row is served sanitized (end-to-end); a plain
local source keeps the pre-phase key set (both keys ABSENT — never
``null``).
PATCH /api/documents/summary (phase 57, task 01) — the admin summary
editor, on the same DB-backed fixtures:
@@ -27,6 +33,7 @@ editor, on the same DB-backed fixtures:
"""
from __future__ import annotations
from collections.abc import Iterator
from datetime import UTC, datetime
import pytest
@@ -37,7 +44,7 @@ from sqlalchemy.orm import Session
import app.api.docs as docs_api
from app.config import get_settings
from app.main import app as fastapi_app
from app.models import Chunk, Document
from app.models import Chunk, Document, GitSource
from app.rag.llm import EmbeddingError
from tests.conftest import ADMIN_PASSWORD
from tests.fakes import FakeEmbedder
@@ -571,6 +578,131 @@ def test_content_format_from_suffix(client, db) -> None:
db.commit()
# ---------------------------------------------------------------------------
# Phase 145 (task 03) — the content payload's git origin fields
# (the task-01 resolver's output, end-to-end over the real registry)
# ---------------------------------------------------------------------------
@pytest.fixture()
def origin_registry(db) -> Iterator[None]:
"""A FRESH ``git_sources`` table around the origin tests: the
resolver reads EVERY ``kind='git'`` row, so a controlled registry
is required (truncated before and after — the
``test_agent_tools.py`` registry pattern)."""
db.execute(text("TRUNCATE git_sources"))
db.commit()
yield
db.execute(text("TRUNCATE git_sources"))
db.commit()
def test_content_carries_origin_url_for_a_seeded_git_row(
client, db, origin_registry
) -> None:
"""A ``kind='git'`` row whose URL basename matches the document's
source → the payload carries ``origin_url`` (the sanitized clone
URL). The row pins no branch (NULL = the default branch) →
``origin_branch`` is ABSENT from the JSON (not ``null`` — the
omission pinned on the wire); every pre-phase key stands."""
_seed_doc(db)
db.add(GitSource(url="https://github.com/reese/Homelab.git", kind="git"))
db.commit()
try:
r = client.get(
"/api/documents/content",
params={"source": "Homelab", "path": "kubernetes.md"},
)
assert r.status_code == 200
body = r.json()
assert set(body) == {
"source", "path", "title", "format", "summary", "created_at",
"content", "indexed_at", "chunks", "is_image", "origin_url",
}
assert body["origin_url"] == "https://github.com/reese/Homelab.git"
assert "origin_branch" not in body # absent — never null
finally:
db.execute(text("TRUNCATE chunks, documents"))
db.commit()
def test_content_carries_origin_branch_for_a_pinned_row(
client, db, origin_registry
) -> None:
"""A pinned row (phase 135 ``branch``) also carries
``origin_branch`` — the value the viewer's title attribute shows."""
_seed_doc(db)
db.add(
GitSource(
url="https://github.com/reese/Homelab.git", kind="git", branch="docs"
)
)
db.commit()
try:
r = client.get(
"/api/documents/content",
params={"source": "Homelab", "path": "kubernetes.md"},
)
body = r.json()
assert body["origin_url"] == "https://github.com/reese/Homelab.git"
assert body["origin_branch"] == "docs"
finally:
db.execute(text("TRUNCATE chunks, documents"))
db.commit()
def test_content_origin_is_sanitized_end_to_end(client, db, origin_registry) -> None:
"""A legacy row whose credential is still embedded in the stored
URL never leaks it: the payload carries the phase-121 sanitized
URL (the resolver's guarantee, end-to-end — the stored row value
is untouched, the output mask does the work)."""
_seed_doc(db)
db.add(
GitSource(url="https://user:tok@example.com/reese/Homelab.git", kind="git")
)
db.commit()
try:
r = client.get(
"/api/documents/content",
params={"source": "Homelab", "path": "kubernetes.md"},
)
body = r.json()
assert body["origin_url"] == "https://example.com/reese/Homelab.git"
assert "user:tok" not in r.text # the credential stays out of the wire
finally:
db.execute(text("TRUNCATE chunks, documents"))
db.commit()
def test_content_origin_absent_for_a_plain_local_source(
client, db, origin_registry
) -> None:
"""No unambiguous git row (a local row is never an origin, and a
non-matching git row does not count) → BOTH origin keys ABSENT
from the JSON (not ``null`` — the omission pinned on the wire; the
payload is byte-identical to pre-phase)."""
_seed_doc(db)
db.add(GitSource(url="/srv/Homelab", kind="local", path="/srv/Homelab"))
db.add(GitSource(url="https://github.com/reese/OtherRepo.git", kind="git"))
db.commit()
try:
r = client.get(
"/api/documents/content",
params={"source": "Homelab", "path": "kubernetes.md"},
)
assert r.status_code == 200
body = r.json()
assert set(body) == {
"source", "path", "title", "format", "summary", "created_at",
"content", "indexed_at", "chunks", "is_image",
}
assert "origin_url" not in body
assert "origin_branch" not in body
finally:
db.execute(text("TRUNCATE chunks, documents"))
db.commit()
def test_content_format_name_token_extensionless(
client, db, monkeypatch
) -> None:
+330 -2
View File
@@ -269,8 +269,11 @@ def test_agent_tools_names_and_parameters() -> None:
"and `limit` (number of lines) to read exactly that "
"part. Do not re-read a document you have already read "
"in full — its full text is already in your prompt. "
"Call one tool at a time — wait for this result before "
"your next call."
"For a document from a git source, the result also "
"carries an `origin` line (and `branch` when pinned) — "
"include that original source in your answer (see your "
"tools instructions). Call one tool at a time — wait "
"for this result before your next call."
)
# The byte-preserved contracts inside the rewrite, pinned as
# substrings: the combined source/path identity, the
@@ -293,6 +296,17 @@ def test_agent_tools_names_and_parameters() -> None:
"Call one tool at a time — wait for this result before "
"your next call."
) in read["description"]
# Phase 145 (task 02): the origin clause mirrors the TOOLS_SECTION
# rule (the description points, the section teaches — the
# phase-124 show_image "see the instructions" shape): a git-
# sourced document's read result carries an `origin` line (and
# `branch` when pinned); include that original source in the
# answer.
assert (
"For a document from a git source, the result also carries "
"an `origin` line (and `branch` when pinned) — include that "
"original source in your answer (see your tools instructions)."
) in read["description"]
read_params = read["parameters"]
assert read_params["type"] == "object"
# Phase 131 (task 01, LOCKED A2): offset/limit are OPTIONAL —
@@ -3649,6 +3663,320 @@ def test_unknown_tool_name_refused(monkeypatch: pytest.MonkeyPatch) -> None:
assert llm.requests[1][1] == AGENT_TOOLS # rejected → tools stay offered
# ---------- phase 145 (task 01): the origin/branch lines in the read result ----------
#: The fixture git row's URL (basename ``skills`` = the fixture source
#: name) and the four origin outcomes the read-branch matrix runs:
#: git row matched (branch NULL), git row + pinned branch, local source
#: (no git row), two rows sharing the basename (ambiguous — the source
#: name is ``repo`` for the last one).
_GIT_URL = "https://github.com/owner/skills"
_GIT_ROWS = [GitSource(url=_GIT_URL, kind="git")]
_GIT_PINNED_ROWS = [GitSource(url=_GIT_URL, kind="git", branch="release/1.2")]
_LOCAL_ROWS = [GitSource(url="/srv/skills", kind="local", path="/srv/skills")]
_AMBIGUOUS_ROWS = [
GitSource(url="https://a.example/org/repo.git", kind="git"),
GitSource(url="git@b.example:other/repo.git", kind="git"),
]
def _git_rows_factory(rows: list[GitSource]) -> Callable[[], Session]:
"""A db factory whose mock session answers the resolver's query with
the git rows of *rows* (like the real ``kind = 'git'`` WHERE clause
— the seeded local rows are filtered out, proving they are never
candidates): the REAL ``origin_for_source`` runs in the read branch,
no database."""
git_rows = [row for row in rows if row.kind == "git"]
def _factory() -> Session:
mock = cast("Session", MagicMock())
mock.__enter__ = MagicMock(return_value=mock)
mock.__exit__ = MagicMock(return_value=False)
mock.scalars = MagicMock(return_value=MagicMock(all=MagicMock(return_value=git_rows)))
return mock
return _factory
def _one_read(
monkeypatch: pytest.MonkeyPatch,
doc: Document,
rows: list[GitSource],
arguments: dict[str, Any],
read_max_chars: int | None = None,
) -> str:
"""One scripted read turn against the REAL resolver; returns the
exact tool-result string (the successful-read bookkeeping is
asserted — the matrix asserts the bytes)."""
monkeypatch.setattr(agent, "find_document", lambda db, source, path: doc)
holder = AgentHolder()
llm = ScriptedLLM(
[ToolCallPiece(id="c1", name="read", arguments=arguments)],
[StreamPiece("content", "ans")],
)
kwargs: dict[str, Any] = {"read_max_chars": read_max_chars} if read_max_chars else {}
asyncio.run(_run(llm, holder, _settings(**kwargs), db_factory=_git_rows_factory(rows)))
assert holder.read_docs == [doc] and holder.tool_calls == 1
return llm.requests[1][0][3]["content"]
def test_origin_lines_plain_read_under_cap(
monkeypatch: pytest.MonkeyPatch, caplog: pytest.LogCaptureFixture
) -> None:
"""Shape 1 (plain read at or under the cap): the origin block rides
IMMEDIATELY AFTER the ``date:`` line — ``Document …: / date: /
origin: / branch: / content`` — in all four origin outcomes; the
first line (the mock's ``_READ_RESULT_PREFIX`` contract) is
untouched, and the local/ambiguous results are byte-identical to
pre-phase (the ambiguity still warns)."""
doc = _doc("skills", "docs/a.md", "A", "A-CONTENT")
plain = {"path": "skills/docs/a.md"}
result = _one_read(monkeypatch, doc, _GIT_ROWS, plain)
assert result == (
"Document skills/docs/a.md:\n"
"date: 2024-06-15\n"
f"origin: {_GIT_URL}\n"
"A-CONTENT"
)
# The E2E mock's header contract: first line, byte-identical.
assert result.startswith("Document ")
assert result.splitlines()[0] == "Document skills/docs/a.md:"
result = _one_read(monkeypatch, doc, _GIT_PINNED_ROWS, plain)
assert result == (
"Document skills/docs/a.md:\n"
"date: 2024-06-15\n"
f"origin: {_GIT_URL}\n"
"branch: release/1.2\n"
"A-CONTENT"
)
result = _one_read(monkeypatch, doc, _LOCAL_ROWS, plain)
assert result == "Document skills/docs/a.md:\ndate: 2024-06-15\nA-CONTENT"
repo_doc = _doc("repo", "docs/a.md", "A", "A-CONTENT")
caplog.clear()
result = _one_read(monkeypatch, repo_doc, _AMBIGUOUS_ROWS, {"path": "repo/docs/a.md"})
assert result == "Document repo/docs/a.md:\ndate: 2024-06-15\nA-CONTENT"
# The ambiguity is logged at WARNING (the agent's INFO per-turn log
# lines are captured too — filter to the resolver's records).
warnings = [
r for r in caplog.records if r.levelname == "WARNING" and r.name == "app.rag.git_sources"
]
assert len(warnings) == 1
assert "repo" in warnings[0].message
def test_origin_lines_truncated_plain_read(
monkeypatch: pytest.MonkeyPatch,
) -> None:
"""Shape 2 (plain read cut at the cap): the origin block sits at the
same position (after ``date:``, before the cut content), the
truncation marker + the pinned notice unchanged, the holder
truncation tuple unchanged (the ``read_truncations`` signal is
untouched)."""
cap = 20
content = "x" * (cap + 1)
doc = _doc("skills", "big.md", "Big", content)
plain = {"path": "skills/big.md"}
result = _one_read(monkeypatch, doc, _GIT_ROWS, plain, read_max_chars=cap)
assert result == (
"Document skills/big.md:\n"
"date: 2024-06-15\n"
f"origin: {_GIT_URL}\n"
+ content[:cap]
+ "\n"
+ TRUNCATION_MARKER
+ "\n"
+ READ_TRUNCATION_NOTICE.format(shown=cap, total=len(content))
)
result = _one_read(monkeypatch, doc, _GIT_PINNED_ROWS, plain, read_max_chars=cap)
assert result == (
"Document skills/big.md:\n"
"date: 2024-06-15\n"
f"origin: {_GIT_URL}\n"
"branch: release/1.2\n"
+ content[:cap]
+ "\n"
+ TRUNCATION_MARKER
+ "\n"
+ READ_TRUNCATION_NOTICE.format(shown=cap, total=len(content))
)
result = _one_read(monkeypatch, doc, _LOCAL_ROWS, plain, read_max_chars=cap)
assert result == (
"Document skills/big.md:\n"
"date: 2024-06-15\n"
+ content[:cap]
+ "\n"
+ TRUNCATION_MARKER
+ "\n"
+ READ_TRUNCATION_NOTICE.format(shown=cap, total=len(content))
)
repo_doc = _doc("repo", "big.md", "Big", content)
result = _one_read(
monkeypatch, repo_doc, _AMBIGUOUS_ROWS, {"path": "repo/big.md"}, read_max_chars=cap
)
assert result == (
"Document repo/big.md:\n"
"date: 2024-06-15\n"
+ content[:cap]
+ "\n"
+ TRUNCATION_MARKER
+ "\n"
+ READ_TRUNCATION_NOTICE.format(shown=cap, total=len(content))
)
def test_origin_lines_range_read_under_cap(
monkeypatch: pytest.MonkeyPatch, caplog: pytest.LogCaptureFixture
) -> None:
"""Shape 3 (range read at or under the cap): the origin block rides
between the ``date:`` line and the ``lines:`` header (the same
position as in the plain shapes); the local/ambiguous results stay
byte-identical to pre-phase."""
doc = _doc("skills", "b.md", "B", "one\ntwo\nthree\nfour\nfive")
ranged = {"path": "skills/b.md", "offset": 2, "limit": 2}
result = _one_read(monkeypatch, doc, _GIT_ROWS, ranged)
assert result == (
"Document skills/b.md:\n"
"date: 2024-06-15\n"
f"origin: {_GIT_URL}\n"
"lines: 2-3 of 5\n"
"two\nthree"
)
result = _one_read(monkeypatch, doc, _GIT_PINNED_ROWS, ranged)
assert result == (
"Document skills/b.md:\n"
"date: 2024-06-15\n"
f"origin: {_GIT_URL}\n"
"branch: release/1.2\n"
"lines: 2-3 of 5\n"
"two\nthree"
)
result = _one_read(monkeypatch, doc, _LOCAL_ROWS, ranged)
assert result == "Document skills/b.md:\ndate: 2024-06-15\nlines: 2-3 of 5\ntwo\nthree"
repo_doc = _doc("repo", "b.md", "B", "one\ntwo\nthree\nfour\nfive")
caplog.clear()
result = _one_read(
monkeypatch, repo_doc, _AMBIGUOUS_ROWS, {"path": "repo/b.md", "offset": 2, "limit": 2}
)
assert result == "Document repo/b.md:\ndate: 2024-06-15\nlines: 2-3 of 5\ntwo\nthree"
# The ambiguity is logged at WARNING (the agent's INFO per-turn log
# lines are captured too — filter to the resolver's records).
warnings = [
r for r in caplog.records if r.levelname == "WARNING" and r.name == "app.rag.git_sources"
]
assert len(warnings) == 1
assert "repo" in warnings[0].message
def test_origin_lines_truncated_range_read(
monkeypatch: pytest.MonkeyPatch,
) -> None:
"""Shape 4 (range read whose SLICE exceeds the cap): the origin block
at the same position; the cap on the slice + the shared marker + the
pinned notice (``{total}`` = the SLICE's length) all unchanged."""
cap = 40
doc = _doc("skills", "c.md", "C", "a" * 30 + "\n" + "b" * 30 + "\n" + "c" * 30)
slice_text = "a" * 30 + "\n" + "b" * 30 # the requested lines 1-2 (61 chars)
ranged = {"path": "skills/c.md", "offset": 1, "limit": 2}
result = _one_read(monkeypatch, doc, _GIT_ROWS, ranged, read_max_chars=cap)
assert result == (
"Document skills/c.md:\n"
"date: 2024-06-15\n"
f"origin: {_GIT_URL}\n"
"lines: 1-2 of 3\n"
+ slice_text[:cap]
+ "\n"
+ TRUNCATION_MARKER
+ "\n"
+ READ_TRUNCATION_NOTICE.format(shown=cap, total=len(slice_text))
)
result = _one_read(monkeypatch, doc, _GIT_PINNED_ROWS, ranged, read_max_chars=cap)
assert result == (
"Document skills/c.md:\n"
"date: 2024-06-15\n"
f"origin: {_GIT_URL}\n"
"branch: release/1.2\n"
"lines: 1-2 of 3\n"
+ slice_text[:cap]
+ "\n"
+ TRUNCATION_MARKER
+ "\n"
+ READ_TRUNCATION_NOTICE.format(shown=cap, total=len(slice_text))
)
result = _one_read(monkeypatch, doc, _LOCAL_ROWS, ranged, read_max_chars=cap)
assert result == (
"Document skills/c.md:\n"
"date: 2024-06-15\n"
"lines: 1-2 of 3\n"
+ slice_text[:cap]
+ "\n"
+ TRUNCATION_MARKER
+ "\n"
+ READ_TRUNCATION_NOTICE.format(shown=cap, total=len(slice_text))
)
repo_doc = _doc("repo", "c.md", "C", "a" * 30 + "\n" + "b" * 30 + "\n" + "c" * 30)
result = _one_read(
monkeypatch, repo_doc, _AMBIGUOUS_ROWS, {"path": "repo/c.md", "offset": 1, "limit": 2},
read_max_chars=cap,
)
assert result == (
"Document repo/c.md:\n"
"date: 2024-06-15\n"
"lines: 1-2 of 3\n"
+ slice_text[:cap]
+ "\n"
+ TRUNCATION_MARKER
+ "\n"
+ READ_TRUNCATION_NOTICE.format(shown=cap, total=len(slice_text))
)
def test_origin_lines_image_doc_keeps_marker_position(
monkeypatch: pytest.MonkeyPatch,
) -> None:
"""The image-doc shape keeps its phase-122 pin: the marker stays
IMMEDIATELY ABOVE the content, and the origin lines ride ABOVE the
marker (``Document …: / date: / origin: / branch: / marker /
content``); a local image doc is byte-identical to pre-phase."""
doc = _doc("skills", "img.png", "Img", "IMG-DESC")
doc.is_image = True
plain = {"path": "skills/img.png"}
result = _one_read(monkeypatch, doc, _GIT_PINNED_ROWS, plain)
assert result == (
"Document skills/img.png:\n"
"date: 2024-06-15\n"
f"origin: {_GIT_URL}\n"
"branch: release/1.2\n"
f"{agent.IMAGE_DOC_MARKER}\n"
"IMG-DESC"
)
doc = _doc("skills", "img.png", "Img", "IMG-DESC")
doc.is_image = True
result = _one_read(monkeypatch, doc, _LOCAL_ROWS, plain)
assert result == (
"Document skills/img.png:\ndate: 2024-06-15\n"
f"{agent.IMAGE_DOC_MARKER}\n"
"IMG-DESC"
)
# ---------- grep (the phase-68 A5 contract under the new name) ----------
+362 -8
View File
@@ -20,9 +20,29 @@ Frontend side:
* phase 26 — the shared ``renderDocument`` export in ``document.js``, the
import-safe page guard, the modal skeleton on chat + Sources, and the
modal module's close/focus/URL contract.
Phase 145 (task 03) — the content payload's git origin fields + the
``renderDocument`` "Original" anchor (modal AND /document.html, one
site):
* the ``DocContent`` omission rule — both origin keys ABSENT on the
wire when ``None`` (never ``null``; non-git payload byte-identical);
* the content endpoint (stubbed session) — a matching ``kind='git'``
row adds ``origin_url`` (+ ``origin_branch`` for a pinned row), a
legacy embedded-credential row is served sanitized, and local /
ambiguous sources keep the pre-phase key set;
* the anchor itself (source pins + a node behavioral check over a
minimal fake DOM) — ``createElement("a")``, the full URL verbatim as
``href`` AND visible text (the copyable-link contract),
``target="_blank" rel="noopener"``, the ``title`` = full URL (+
`` — branch: <name>`` when pinned), inserted AFTER the path code and
BEFORE the Created badge (absent → the pre-phase six children), no
innerHTML/outerHTML (the module's ONE innerHTML survives), and the
``.doc-origin`` CSS mirroring the ``.doc-path`` ellipsis idiom with
the ``.doc-link`` affordance (+ the ≤640px squeeze).
"""
from __future__ import annotations
import json
import re
import shutil
import subprocess
@@ -35,7 +55,8 @@ from fastapi.testclient import TestClient
from app.api.docs import doc_format
from app.db import get_db
from app.main import create_app
from app.models import Document
from app.models import Document, GitSource
from app.schemas import DocContent
from tests.conftest import ADMIN_PASSWORD
FRONTEND = Path(__file__).resolve().parents[2] / "frontend"
@@ -126,22 +147,39 @@ class _FakeResult:
return self._row
class _FakeScalars:
"""The resolver's ``db.scalars(...).all()`` — only the git rows
ever reach the caller (the ``kind = 'git'`` WHERE clause is
emulated, the task-01 ``_GitOnlySession`` pattern)."""
def __init__(self, rows: tuple[GitSource, ...]) -> None:
self._rows = [row for row in rows if row.kind == "git"]
def all(self) -> list[GitSource]:
return self._rows
class _FakeSession:
def __init__(self, row: object) -> None:
def __init__(self, row: object, git_rows: tuple[GitSource, ...] = ()) -> None:
self._row = row
self._git_rows = git_rows
def execute(self, _stmt: object) -> _FakeResult:
return _FakeResult(self._row)
def scalars(self, _stmt: object) -> _FakeScalars:
return _FakeScalars(self._git_rows)
def _client_with_row(row: object) -> TestClient:
def _client_with_row(row: object, git_rows: tuple[GitSource, ...] = ()) -> TestClient:
"""Fresh app whose ``get_db`` dependency is a stub returning ``row``
(``None`` → no matching document row). Phase 79: the endpoint is
user-gated, so the client signs in as the admin first — these tests
pin the CONTENT mapping (200/404/422), not the auth contract (which
``test_auth_api.py`` pins)."""
(``None`` → no matching document row) and, for the phase-145
origin lookup, *git_rows* (default: no rows — the non-git corner).
Phase 79: the endpoint is user-gated, so the client signs in as
the admin first — these tests pin the CONTENT mapping (200/404/
422), not the auth contract (which ``test_auth_api.py`` pins)."""
app = create_app()
app.dependency_overrides[get_db] = lambda: _FakeSession(row)
app.dependency_overrides[get_db] = lambda: _FakeSession(row, git_rows)
client = TestClient(app)
r = client.post("/api/login", json={"password": ADMIN_PASSWORD})
assert r.status_code == 204, f"admin login failed: {r.status_code} {r.text}"
@@ -217,6 +255,322 @@ def test_content_requires_both_params() -> None:
assert client.get("/api/documents/content", params={"source": "Homelab"}).status_code == 422
# ---------------------------------------------------------------------------
# Phase 145 (task 03) — the content payload's git origin fields + the
# renderDocument "Original" anchor (modal AND /document.html, one site)
# ---------------------------------------------------------------------------
def _text_doc() -> Document:
"""The content-endpoint stub's document row (a markdown doc — no
summary, the pre-phase shape)."""
doc = Document(
source="Homelab",
path="notes/deep mark.md",
full_path="/tmp/deep mark.md",
title="Deep Mark",
content="# Deep Mark\n\nbody",
content_hash="f" * 64,
is_image=False, # phase 122: the stub session never applies defaults
)
doc.indexed_at = datetime(2026, 8, 22, 1, 2, 3, tzinfo=UTC)
doc.created_at = datetime(2026, 8, 20, 9, 0, 0, tzinfo=UTC)
return doc
def _text_doc_content(
origin_url: str | None = None, origin_branch: str | None = None
) -> DocContent:
"""One text doc's content payload (the pre-phase shape) with the
phase-145 origin fields optionally set."""
return DocContent(
source="S",
path="a.md",
title="A",
format="md",
created_at="2026-01-01T00:00:00+00:00",
content="body",
indexed_at="2026-01-01T00:00:00+00:00",
chunks=1,
origin_url=origin_url,
origin_branch=origin_branch,
)
def test_doc_content_origin_fields_absent_on_the_wire_when_unset() -> None:
"""The model-level omission rule (the :func:`_drop_absent_share_url`
precedent, the ``image_url`` shape): a non-git document (no
``origin_url``) serializes with BOTH origin keys ABSENT — never
``null`` — so the payload is byte-identical to pre-phase. A
sanitized-URL-only payload (an unpinned row) drops
``origin_branch``; a pinned row keeps both keys."""
dumped = _text_doc_content().model_dump()
assert "origin_url" not in dumped # absent — never null
assert "origin_branch" not in dumped # absent — never null
dumped = _text_doc_content(origin_url="https://host/x.git").model_dump()
assert dumped["origin_url"] == "https://host/x.git"
assert "origin_branch" not in dumped # NULL = the default branch — nothing to show
dumped = _text_doc_content(
origin_url="https://host/x.git", origin_branch="docs"
).model_dump()
assert dumped["origin_branch"] == "docs"
def test_content_origin_url_carried_for_a_matching_git_row() -> None:
"""The endpoint (stubbed session, no database): a ``kind='git'``
row whose URL basename matches the document's source adds
``origin_url`` — the sanitized clone URL — to the payload, and
every pre-phase key stands untouched. The row pins no branch →
``origin_branch`` is ABSENT (never ``null``)."""
row = GitSource(url="https://github.com/reese/Homelab.git", kind="git")
with _client_with_row((_text_doc(), 3), git_rows=(row,)) as client:
r = client.get(
"/api/documents/content",
params={"source": "Homelab", "path": "notes/deep mark.md"},
)
assert r.status_code == 200
body = r.json()
assert set(body) == {
"source", "path", "title", "format", "summary", "created_at",
"content", "indexed_at", "chunks", "is_image", "origin_url",
}
assert body["origin_url"] == "https://github.com/reese/Homelab.git"
assert "origin_branch" not in body # absent — never null (unpinned row)
def test_content_origin_branch_carried_for_a_pinned_row() -> None:
"""A pinned row (phase 135 ``branch``) also carries
``origin_branch`` — the branch the title attribute will show."""
row = GitSource(
url="https://github.com/reese/Homelab.git", kind="git", branch="docs"
)
with _client_with_row((_text_doc(), 3), git_rows=(row,)) as client:
r = client.get(
"/api/documents/content",
params={"source": "Homelab", "path": "notes/deep mark.md"},
)
body = r.json()
assert body["origin_url"] == "https://github.com/reese/Homelab.git"
assert body["origin_branch"] == "docs"
def test_content_origin_url_sanitizes_a_legacy_embedded_credential_row() -> None:
"""End-to-end (stubbed): a legacy row whose credential is still
embedded in the stored URL never leaks it — the payload carries
the phase-121 sanitized URL (the resolver's guarantee, one mapping
for the whole app)."""
row = GitSource(url="https://user:tok@example.com/reese/Homelab.git", kind="git")
with _client_with_row((_text_doc(), 3), git_rows=(row,)) as client:
r = client.get(
"/api/documents/content",
params={"source": "Homelab", "path": "notes/deep mark.md"},
)
body = r.json()
assert body["origin_url"] == "https://example.com/reese/Homelab.git"
assert "user:tok" not in r.text # the credential stays out of the wire
def test_content_origin_fields_absent_for_local_and_ambiguous_sources() -> None:
"""No git row (a plain local source) OR two rows sharing the
basename (ambiguous — never guess) → both origin keys ABSENT from
the JSON (not ``null`` — the omission pinned on the wire; the
payload is byte-identical to pre-phase, the ``test_content_known_
pair_maps_to_doc_content`` key-set pin)."""
local_only = (_text_doc(), 3)
with _client_with_row(local_only) as client: # no git rows at all
r = client.get(
"/api/documents/content",
params={"source": "Homelab", "path": "notes/deep mark.md"},
)
assert r.status_code == 200
assert "origin_url" not in r.json()
assert "origin_branch" not in r.json()
ambiguous = (
GitSource(url="https://a.example/org/Homelab.git", kind="git"),
GitSource(url="git@b.example:other/Homelab.git", kind="git"),
)
with _client_with_row(local_only, git_rows=ambiguous) as client:
r = client.get(
"/api/documents/content",
params={"source": "Homelab", "path": "notes/deep mark.md"},
)
assert r.status_code == 200
assert "origin_url" not in r.json()
assert "origin_branch" not in r.json()
# ---------- the renderDocument "Original" anchor (the shared core) ----------
def _fn(js: str, name: str) -> str:
"""The source of a top-level function via balanced-brace counting
(the ``test_date_editor.py`` pattern)."""
start = js.find(f"function {name}(")
assert start != -1, f"{name}() must exist in document.js"
i = js.find("(", start)
depth = 0
close = i
while i < len(js):
if js[i] == "(":
depth += 1
elif js[i] == ")":
depth -= 1
if depth == 0:
close = i
break
i += 1
brace = js.find("{", close)
depth = 0
for j in range(brace, len(js)):
if js[j] == "{":
depth += 1
elif js[j] == "}":
depth -= 1
if depth == 0:
return js[start : j + 1]
raise AssertionError(f"unbalanced braces in {name}()")
def test_origin_anchor_is_a_copyable_external_link() -> None:
"""The anchor (house-style source contract): ``createElement("a")``
(the never-build-HTML rule), the ``doc-origin`` class, ``href =
doc.origin_url``, ``target="_blank" rel="noopener"`` (the modal's
"Full page" escape-hatch idiom), the full URL VERBATIM as the
visible text (the copyable-link contract — no decorative label),
and the ``title`` = the full URL, plus `` — branch: <name>`` when
the payload pins a branch (the pathCode precision idiom)."""
body = _fn(_read(DOCUMENT_JS), "originAnchor")
assert 'document.createElement("a")' in body
assert 'a.className = "doc-origin"' in body
assert "a.href = doc.origin_url" in body
assert 'a.target = "_blank"' in body
assert 'a.rel = "noopener"' in body
assert "a.textContent = doc.origin_url" in body, (
"the visible text is the full URL verbatim (copy-pasteable)"
)
branch_title = 'doc.origin_url + " — branch: " + doc.origin_branch'
assert 'doc.origin_branch ? ' + branch_title + " : doc.origin_url" in body, (
"the title carries the full URL (+ the pinned branch when present)"
)
# The never-build-HTML shape: no innerHTML/outerHTML in the new code.
code = re.sub(r"//.*?$|/\*.*?\*/", "", body, flags=re.S | re.M)
assert "innerHTML" not in code and "outerHTML" not in code
@pytest.mark.skipif(not HAVE_NODE, reason="node not available")
def test_origin_anchor_behaves_for_branch_and_no_branch_variants() -> None:
"""Behavioral check (node, minimal fake DOM) of the real anchor
builder: the exact ``href``/``target``/``rel``/``textContent``/
``title`` for a git doc, branch and no-branch variants."""
fn = _fn(_read(DOCUMENT_JS), "originAnchor")
fake_dom = (
"globalThis.document = { createElement: (tag) => {"
' const el = { tag, className: "", href: "", target: "", rel: "",'
' textContent: "", setAttribute(k, v) { this[k] = v; } };'
" return el; } };"
)
out = _run_node(
fake_dom
+ "\n" + fn
+ '\nconst g1 = originAnchor({ origin_url: "https://github.com/reese/skills.git",'
' origin_branch: "main" });' + "\n"
+ '\nconst g2 = originAnchor({ origin_url: "https://github.com/reese/skills.git" });' + "\n"
+ "console.log(JSON.stringify(g1));\nconsole.log(JSON.stringify(g2));"
)
j1, j2 = (json.loads(line) for line in out.splitlines())
for j in (j1, j2):
assert j["tag"] == "a"
assert j["className"] == "doc-origin"
assert j["href"] == "https://github.com/reese/skills.git"
assert j["target"] == "_blank"
assert j["rel"] == "noopener"
# The full URL VERBATIM as the visible text (copy-pasteable).
assert j["textContent"] == "https://github.com/reese/skills.git"
# The title: the full URL, + the pinned branch when present.
assert j1["title"] == "https://github.com/reese/skills.git — branch: main"
assert j2["title"] == "https://github.com/reese/skills.git"
def test_origin_anchor_sits_after_the_path_code_before_the_created_badge() -> None:
"""renderDocument (the ONE shared core — modal AND /document.html):
the anchor joins the meta row AFTER the path ``code`` and BEFORE
the Created badge — and ONLY when the payload carries
``origin_url``. Absent (a non-git document — the field simply
missing on the wire) → the gate builds the empty array, the
spread adds nothing, and the meta row's children are EXACTLY the
pre-phase six (the existing badge-order pins pass unchanged)."""
render = _fn(_read(DOCUMENT_JS), "renderDocument")
gate_i = render.find("const originItems = doc.origin_url ? [originAnchor(doc)] : [];")
call_i = render.find("metaEl.replaceChildren(")
path_i = render.find("pathCode,", call_i)
spread_i = render.find("...originItems,", call_i)
created_i = render.find('metaBadge("doc-created"', call_i)
assert -1 < gate_i < call_i < path_i < spread_i < created_i, (
"the anchor gate precedes the row build and lands AFTER the path code, "
"BEFORE the Created badge"
)
# The gate: present only on a truthy origin_url — an ABSENT field
# (undefined) is falsy → [] → the pre-phase children stand.
assert "doc.origin_url ? [originAnchor(doc)] : []" in render
# The spread is the meta row's ONLY conditional child (the six
# pre-phase children are the fixed ones).
assert render[call_i:].count("...") == 1
def test_origin_anchor_keeps_the_module_xss_contract() -> None:
"""The module's ONE innerHTML (the escape-first markdown render)
survives — the new anchor adds no HTML-building of its own
(createElement + properties/textContent only)."""
code = re.sub(r"//.*", "", re.sub(r"/\*.*?\*/", "", _read(DOCUMENT_JS), flags=re.S))
assert code.count("innerHTML") == 1
assert "outerHTML" not in code
def test_doc_origin_css_mirrors_the_doc_path_ellipsis_idiom() -> None:
"""styles.css: the ``.doc-origin`` rule sits next to the
``.doc-path`` family (mono, 0.75rem, the same 26rem ellipsis
budget + 6rem floor) with the ``.doc-link`` affordance
(brand-ink, underline on hover/focus — the existing link
tokens), provenance-cited, and the ≤640px block squeezes it like
the path (16rem — the row's own wrap/clip rules are unchanged:
the anchor is one more inline item)."""
css = _read(FRONTEND / "assets" / "styles.css")
rule_i = css.find(".doc-origin {")
assert rule_i != -1, "styles.css must style .doc-origin"
# Provenance comment directly above the rule (house style).
comment_start = css.rfind("/*", 0, rule_i)
assert -1 < comment_start < rule_i and "Phase 145 (task 03)" in css[comment_start:rule_i]
# Next to the .doc-path family (before .doc-chunks).
assert css.find(".doc-path {") < rule_i < css.find(".doc-chunks {")
rule = css[rule_i : css.find("}", rule_i)]
for decl in (
"font-family: var(--mono)",
"font-size: 0.75rem",
"max-width: 26rem",
"white-space: nowrap",
"overflow: hidden",
"text-overflow: ellipsis",
"min-width: 6rem",
"color: var(--brand-ink)",
"text-decoration: none",
):
assert decl in rule, f".doc-origin must carry {decl!r}"
# The .doc-link hover/focus affordance (the existing link idiom).
assert ".doc-origin:hover, .doc-origin:focus-visible { text-decoration: underline; }" in css
# The ≤640px squeeze mirrors .doc-path's 16rem (same media block).
mobile = css.find("@media (max-width: 640px)")
path_sq = css.find(".doc-path { max-width: 16rem; }", mobile)
origin_sq = css.find(".doc-origin { max-width: 16rem; }", mobile)
close_640 = css.find("@media", mobile + 1)
assert -1 < path_sq < origin_sq < (close_640 if close_640 != -1 else len(css)), (
"the mobile block squeezes the anchor like the path"
)
assert "url(http" not in css and "@import url(" not in css, (
"no CDN (AGENTS.md rule 6)"
)
# ---------------------------------------------------------------------------
# Viewer URL builder — encoded query (spaces/slashes in real paths)
# ---------------------------------------------------------------------------
+121
View File
@@ -179,3 +179,124 @@ def test_alias_env_fallback_unchanged(monkeypatch: pytest.MonkeyPatch) -> None:
assert urls == ["https://env.example/one.git"]
assert origin == "env"
# --- phase 145 (task 01): origin_for_source (the read result's origin) ---
class _GitOnlySession:
"""The origin resolver's session (phase 145, task 01): emulates the
resolver's ``kind = 'git'`` WHERE clause — only the git rows ever
reach the caller, exactly like the real query; the seeded local
rows prove they are never candidates."""
def __init__(self, rows: list[GitSource]) -> None:
self._rows = [row for row in rows if row.kind == "git"]
self.statements: list[Any] = []
def scalars(self, stmt: Any) -> _FakeScalars:
self.statements.append(stmt)
return _FakeScalars(self._rows)
# --- the basename mapping --------------------------------------------------
def test_origin_basename_mapping() -> None:
"""The URL's path component — after the ``://``, or after the ``:``
for scp-style ``git@host:owner/repo`` — trailing ``/`` stripped,
a single trailing ``.git`` suffix stripped, last segment taken. An
``@`` inside the PATH is not userinfo and stays (no split, no
strip)."""
cases = {
"https://github.com/owner/myrepo.git": "myrepo", # the .git suffix
"https://github.com/owner/myrepo": "myrepo", # no suffix, same dir
"https://github.com/owner/myrepo/": "myrepo", # trailing slash
"https://github.com/owner/myrepo.git/": "myrepo", # both
"git@github.com:owner/myrepo.git": "myrepo", # scp-style + suffix
"git@github.com:owner/myrepo": "myrepo", # scp-style, no suffix
"ssh://git@host/owner/myrepo.git": "myrepo", # ssh scheme, @ in the host part
"https://host/a@b/myrepo.git": "myrepo", # @ inside the PATH — not userinfo
}
for url, expected in cases.items():
assert resolver._url_basename(url) == expected, url
# --- the match: sanitize + branch ------------------------------------------
def test_origin_exact_match_is_sanitized() -> None:
"""EXACTLY ONE match → the phase-121 masked URL + the branch — a
legacy embedded-credential row never leaks its secret (the stored
value is untouched; the OUTPUT mask does the work)."""
session = _GitOnlySession(
[GitSource(url="https://user:tok@host/x.git", kind="git", branch=None)]
)
assert resolver.origin_for_source(session, "x") == ("https://host/x.git", None) # pyright: ignore[reportArgumentType]
def test_origin_branch_passthrough() -> None:
"""The branch rides through as-is: pinned → the name (the read
result's ``branch:`` line), NULL → ``None`` (the default branch —
no ``branch:`` line)."""
session = _GitOnlySession(
[GitSource(url="https://host/x.git", kind="git", branch="release/1.2")]
)
assert resolver.origin_for_source(session, "x") == ("https://host/x.git", "release/1.2") # pyright: ignore[reportArgumentType]
def test_origin_local_rows_never_origins() -> None:
"""Local rows are never origins (nothing to link to from the user's
browser): a source that exists ONLY as a local row gets no origin
(zero git candidates); a local + git pair resolves to the git row."""
local = GitSource(url="/srv/repo", kind="local", path="/srv/repo")
session = _GitOnlySession([local]) # the WHERE clause drops it
assert resolver.origin_for_source(session, "repo") is None # pyright: ignore[reportArgumentType]
session = _GitOnlySession(
[local, GitSource(url="https://host/repo.git", kind="git", branch=None)]
)
assert resolver.origin_for_source(session, "repo") == ("https://host/repo.git", None) # pyright: ignore[reportArgumentType]
# --- the no-guess rows: None + WARNING --------------------------------------
def test_origin_zero_match_is_none_with_warning(
caplog: pytest.LogCaptureFixture,
) -> None:
"""No git row's basename matches (a plain local source) → ``None``
+ a WARNING naming the source — the read result stays
byte-identical to pre-phase."""
session = _GitOnlySession([GitSource(url="https://host/other.git", kind="git")])
assert resolver.origin_for_source(session, "repo") is None # pyright: ignore[reportArgumentType]
warnings = [record for record in caplog.records if record.levelname == "WARNING"]
assert len(warnings) == 1
assert "repo" in warnings[0].message
def test_origin_ambiguous_is_none_with_warning(
caplog: pytest.LogCaptureFixture,
) -> None:
"""TWO-OR-MORE rows share the basename → ``None`` + a WARNING naming
the sanitized candidates (never guess — a wrong origin would ride
into every answer about the document). The candidates are logged
SANITIZED: a legacy embedded credential stays out of the log too."""
session = _GitOnlySession(
[
GitSource(url="https://user:tok@a.example/org/repo.git", kind="git"),
GitSource(url="git@b.example:other/repo.git", kind="git"),
]
)
assert resolver.origin_for_source(session, "repo") is None # pyright: ignore[reportArgumentType]
warnings = [record for record in caplog.records if record.levelname == "WARNING"]
assert len(warnings) == 1
assert "https://a.example/org/repo.git" in warnings[0].message
assert "git@b.example:other/repo.git" in warnings[0].message
assert "tok" not in warnings[0].message
+75 -6
View File
@@ -15,6 +15,7 @@ from __future__ import annotations
import hashlib
from app.rag.prompts import (
ORIGIN_RULE,
PERSONA,
SUGGEST_INTRO,
TOOLS_SECTION,
@@ -77,12 +78,18 @@ def test_high_and_low_bases_byte_locked() -> None:
#: form — do not re-read a document read IN FULL (answer from it);
#: a PARTIAL document (a range read, or a truncated read) is not
#: complete in the context until read all — read the other parts with
#: ``offset``/``limit``. Only that sentence moved — the prefix (the
#: ``ls``-clause opening) and the suffix (the discipline-rules
#: ending) survived byte-identical, so they are the same anchors as
#: pre-phase-141.
TOOLS_SECTION_SHA256 = "3671680bbf4e04efe1186690c18d24869e4974ceb29485cedc3e34a9286bf3b2"
TOOLS_SECTION_LEN = 2656
#: ``offset``/``limit``. RE-CUT again for phase 145 (task 02): the
#: ``read`` clause gains the original-source rule — :data:`ORIGIN_RULE`,
#: the one sentence-group appended at the ``read`` clause's end (the
#: truncation workflow sentence) and BEFORE the ``grep`` clause begins
#: (the ``origin:`` line is a read-result fact; the rule binds the
#: ANSWER that uses what the read added). Only that sentence-group was
#: inserted — the prefix (the ``ls``-clause opening) and the suffix
#: (the discipline-rules ending) survived byte-identical, so they are
#: the same anchors as pre-phase-145; the pre-phase-145 sha/length are
#: kept below for the diff-style pin.
TOOLS_SECTION_SHA256 = "41a84885be98d4e15b95205694d59d364296a30e27758d39ae737881d5bacaa7"
TOOLS_SECTION_LEN = 3279
TOOLS_SECTION_PREFIX = (
"<tools>\n"
"You may extend your context with three tools. `ls` lists the "
@@ -94,6 +101,33 @@ TOOLS_SECTION_SUFFIX = (
"you need.\n</tools>"
)
#: The pre-phase-145 anchors — the section BEFORE :data:`ORIGIN_RULE` was
#: inserted. Kept for the diff-style pin: the ONLY byte change is the
#: rule's insertion, so removing the rule reproduces these exact bytes.
TOOLS_SECTION_PRE_PHASE_145_SHA256 = (
"3671680bbf4e04efe1186690c18d24869e4974ceb29485cedc3e34a9286bf3b2"
)
TOOLS_SECTION_PRE_PHASE_145_LEN = 2656
#: Phase 145 (task 02) — the original-source rule the ``read`` teaching
#: carries is :data:`app.rag.prompts.ORIGIN_RULE` (imported above, LOCKED
#: verbatim). :data:`TOOLS_SECTION` embeds that exact sentence-group at
#: the ``read`` clause's end (the truncation workflow sentence) and
#: BEFORE the ``grep`` clause begins. A read result for a git-sourced
#: document carries an ``origin:`` line (and a ``branch:`` line when a
#: branch is pinned) — the read contract's fact — so when the answer uses
#: such a document it includes the original source in the answer: a link
#: to the repository (the ``origin`` line) with the document's path (the
#: ``Document …`` header) named and the pinned branch when one is shown,
#: as the repository's file-view URL where the model can tell it
#: (``<repo-url>/blob/<branch>/<path>``), or the repository URL plus the
#: path as plain text otherwise. The rule is CONDITIONAL on the ``origin``
#: line's presence — a local-sourced document's read result carries none,
#: so the rule adds nothing to those answers (the model has no URL to
#: cite; it must not invent one — the sentence's conditioning does that
#: job).
def test_tools_section_byte_locked() -> None:
"""The ``<tools>`` teaching is LOCKED verbatim too (the E2E mock keys
@@ -105,6 +139,41 @@ def test_tools_section_byte_locked() -> None:
assert TOOLS_SECTION.endswith(TOOLS_SECTION_SUFFIX)
def test_tools_section_origin_rule_verbatim_at_its_position() -> None:
"""Phase 145 (task 02): the original-source rule is LOCKED verbatim
and sits at its EXACT position — the ``read`` clause's end (the
truncation workflow sentence) and BEFORE the ``grep`` clause begins.
The sentence-group is pinned byte-for-byte, and the surrounding
boundary (the truncation sentence's tail → the rule → the ``grep``
clause's opening) is pinned as one contiguous substring, so a move
or re-spelling of ANY byte (the rule's or the boundary's) fails."""
# Verbatim presence.
assert ORIGIN_RULE in TOOLS_SECTION
assert TOOLS_SECTION.count(ORIGIN_RULE) == 1 # exactly once
# Exact position: after the truncation sentence, before the grep
# clause — one contiguous span, no drift at either seam.
assert (
"it searches the whole document. "
+ ORIGIN_RULE
+ " `grep` locates an exact string"
) in TOOLS_SECTION
def test_tools_section_origin_rule_is_the_only_byte_change() -> None:
"""Phase 145 (task 02) diff-style pin: the section's OTHER bytes are
byte-identical to pre-phase — the pre-phase section with ONLY the new
sentence-group inserted. Removing the rule (and the single space
that now separates it from the ``grep`` clause) reproduces the
pre-phase-145 section byte-for-byte (sha + length), and the length
delta is exactly the rule's length plus that one inserted space."""
assert len(TOOLS_SECTION) == (
TOOLS_SECTION_PRE_PHASE_145_LEN + len(ORIGIN_RULE) + 1
)
pre_phase = TOOLS_SECTION.replace(ORIGIN_RULE + " ", "")
assert len(pre_phase) == TOOLS_SECTION_PRE_PHASE_145_LEN
assert _sha256(pre_phase) == TOOLS_SECTION_PRE_PHASE_145_SHA256
# ---------- the LOW prompt's locked DEFLECT_MODE body ----------
#: The ``DEFLECT_MODE`` body exactly as it was pre-phase-112 (the
+38
View File
@@ -48,6 +48,7 @@ import pytest
from app.config import Settings
from app.models import Document
from app.rag.prompts import (
ORIGIN_RULE,
PERSONA,
SHOW_IMAGE_CLAUSE,
SUGGEST_INTRO,
@@ -463,6 +464,43 @@ def test_tools_section_phase72_clauses_in_high_prompt_not_low() -> None:
assert "including the source name" not in low
def test_tools_section_phase145_origin_rule() -> None:
"""Phase 145 (task 02): the ``<tools>`` ``read`` clause carries the
original-source rule (LOCKED verbatim in :data:`ORIGIN_RULE`) — when
the answer uses a document whose read result carries an ``origin:``
line, it includes the original source (a link to the repository
from the origin line, with the document's path and the pinned branch
where shown) so the user opens the complete original in one step.
The rule rides the HIGH prompt (``<tools>`` is HIGH-only) and never
the LOW/deflection prompt. The citation discipline is UNCHANGED —
the rule extends the ANSWERS of cited documents; it does not widen
the cite set (the "cite only what you read" sentence stays
byte-identical — pinned in ``test_prompt_lock``)."""
# The rule is present verbatim in the constant, exactly once, and at
# its exact position (the read clause's end, before the grep clause).
assert TOOLS_SECTION.count(ORIGIN_RULE) == 1
assert (
"it searches the whole document. "
+ ORIGIN_RULE
+ " `grep` locates an exact string"
) in TOOLS_SECTION
# It rides the built HIGH prompt with the rest of the section.
doc = _doc("kubernetes.md", "Talos Linux on three nodes.", "Kubernetes")
high = build_high_prompt([doc])
assert ORIGIN_RULE in high
# ...and never the LOW/deflection prompt (``<tools>`` is HIGH-only).
for low in (build_deflect_prompt(["T1"]), build_deflect_prompt([])):
assert ORIGIN_RULE not in low
# The citation discipline is UNCHANGED: the rule does not widen the
# cite set (the "cite only what you read" sentence is untouched).
assert (
"Cite only the document(s) you read — or, if you answered from a "
"suggested summary without reading it, cite that suggested "
"document — never a document you neither read nor used."
) in high
assert "Cite only the document(s) you read" in SUGGEST_INTRO
def test_documents_section_leads_with_the_suggest_intro() -> None:
"""Phase 118, task 03: the ``<documents>`` section leads with the
start-here :data:`SUGGEST_INTRO` line BEFORE the first block (the