phase: 145_git_source_links
**Phase 145 final verification pass — all green, no code changes needed** - **Verified (no defects found):** resolver `origin_for_source` + `_url_basename` (scp/`.git`/trailing-slash, sanitize, ambiguity WARNING); `origin:`/`branch:` lines in all four read shapes via shared `_origin_lines`; HIGH `<tools>` origin rule (verbatim + only-byte-change pins); `DocContent.origin_url/origin_branch` (absent-never-null serializer); `renderDocument` "Original" anchor (copyable verbatim URL, `_blank`/`noopener`, branch in title); mock scripted origin-link flow (disjoint trigger — grep-verified vs all other E2E files) - **Tests:** `uv run pytest --cov=app --cov-report=term-missing` → 3402 passed, `app/` coverage **99%** (>90% gate); `uv run pytest tests/e2e/test_git_source_links.py -v --no-cov` → **4 passed** in isolation; `uv run ruff check .` → clean; `uv run pyright` → 0 errors, 0 warnings - **Completion criteria:** (1) git read results carry origin/branch, local byte-identical — ✅ (unit pins + E2E control); (2) HIGH `<tools>` rule pinned, LOW byte-identical — ✅; (3) modal AND /document.html show Original link, branch in label when pinned, nothing for local — ✅; (4) full suite green + coverage + E2E + lint/types — ✅; (5) no completed-phase behavior change (read header line 1, mock prefix, done frame/SourceRef untouched, LOW sha) — ✅ - **Extra regression check:** re-ran shared-mock E2E suites `test_read_line_ranges.py` (2 passed) and `test_document_viewer.py` (8 passed) — no cross-suite regressions from the `mock_llm.py` addition - **Documentation:** README.md reviewed in full (env table ⊇ `.env.example`, pages/routes, config defaults, scripts, 17-color palette, 300 s guard, 2 s sync poll, phase-145 provenance section matches implementation) and AGENTS.md (72rem container, no-CDN test, smoke test, gitignore, gates) — **verified consistent, no changes needed** - **Notable:** phase-116 screenshot PNG was regenerated during this phase's E2E runs (byte re-encode only, visually identical, correct no-anchor local-source meta row) — left in tree for the harness commit - **Next pending phase:** `146_search_tool_scope` (then 147, 148, 149)
This commit is contained in:
+44
-3
@@ -280,7 +280,7 @@ from sqlalchemy.orm import Session
|
||||
from app.config import Settings
|
||||
from app.models import Document, FolderSummary
|
||||
from app.rag.folder_summaries import folder_of
|
||||
from app.rag.git_sources import effective_sources
|
||||
from app.rag.git_sources import effective_sources, origin_for_source
|
||||
from app.rag.llm import (
|
||||
LLMClient,
|
||||
LLMError,
|
||||
@@ -362,8 +362,11 @@ AGENT_TOOLS: list[dict[str, Any]] = [
|
||||
"first line) and `limit` (number of lines) to read "
|
||||
"exactly that part. Do not re-read a document you have "
|
||||
"already read in full — its full text is already in "
|
||||
"your prompt. Call one tool at a time — wait for this "
|
||||
"result before your next call."
|
||||
"your prompt. For a document from a git source, the "
|
||||
"result also carries an `origin` line (and `branch` "
|
||||
"when pinned) — include that original source in your "
|
||||
"answer (see your tools instructions). Call one tool at "
|
||||
"a time — wait for this result before your next call."
|
||||
),
|
||||
"parameters": {
|
||||
"type": "object",
|
||||
@@ -1272,6 +1275,32 @@ def _resolve_path(db: Session, combined: str) -> tuple[Document | None, str, str
|
||||
return find_document(db, source, path), source, path
|
||||
|
||||
|
||||
def _origin_lines(origin: tuple[str, str | None] | None) -> str:
|
||||
"""Phase 145 (task 01) — the read result's origin lines, shared by
|
||||
the four result builders (the lines are identical in every shape).
|
||||
|
||||
``None`` (the document's source has no unambiguous ``kind='git'``
|
||||
row — a local source, zero matches, or an ambiguous basename) →
|
||||
``""``: the result stays byte-identical to pre-phase. Otherwise
|
||||
one ``origin: {url}`` line (the sanitized repository URL — the
|
||||
phase-121 mask, so a legacy embedded-credential row's secret never
|
||||
reaches the prompt) and, when a branch is pinned (non-NULL), one
|
||||
``branch: {branch}`` line. The builders insert the block
|
||||
IMMEDIATELY AFTER the ``date:`` line and BEFORE the image marker
|
||||
(the marker keeps its phase-122 position relative to the content,
|
||||
the origin lines ride above it); the FIRST line
|
||||
(``Document {source}/{path}:``) is untouched — the E2E mock's
|
||||
``_READ_RESULT_PREFIX`` header contract.
|
||||
"""
|
||||
if origin is None:
|
||||
return ""
|
||||
url, branch = origin
|
||||
lines = f"origin: {url}\n"
|
||||
if branch is not None:
|
||||
lines += f"branch: {branch}\n"
|
||||
return lines
|
||||
|
||||
|
||||
def all_documents(db: Session) -> list[Document]:
|
||||
"""Every indexed document (full rows), ordered by ``(source, path)``
|
||||
— catalog order.
|
||||
@@ -1728,6 +1757,14 @@ def _execute_tool(
|
||||
if (doc.source, doc.path) not in known:
|
||||
holder.read_docs.append(doc)
|
||||
holder.tool_calls += 1
|
||||
# Phase 145 (task 01): the git-sourced document's repository
|
||||
# origin — computed ONCE per successful read, BEFORE the
|
||||
# ranged/truncated branching (the lines are identical in all
|
||||
# four result shapes, which each render the shared block via
|
||||
# :func:`_origin_lines`). ``None`` (local source, no git row,
|
||||
# ambiguous basename) → no lines: the result stays
|
||||
# byte-identical to pre-phase.
|
||||
origin = origin_for_source(db, doc.source)
|
||||
# Phase 122 (task 05): an image document's content IS the vision
|
||||
# description — the marker line (a third line between the
|
||||
# byte-identical header/date lines and the text) tells the model
|
||||
@@ -1775,6 +1812,7 @@ def _execute_tool(
|
||||
return (
|
||||
f"Document {doc.source}/{doc.path}:\n"
|
||||
f"date: {doc.created_at:%Y-%m-%d}\n"
|
||||
f"{_origin_lines(origin)}"
|
||||
f"{marker}"
|
||||
f"{doc.content[:cap]}\n"
|
||||
f"{TRUNCATION_MARKER}\n"
|
||||
@@ -1792,6 +1830,7 @@ def _execute_tool(
|
||||
return (
|
||||
f"Document {doc.source}/{doc.path}:\n"
|
||||
f"date: {doc.created_at:%Y-%m-%d}\n"
|
||||
f"{_origin_lines(origin)}"
|
||||
f"{marker}"
|
||||
f"{doc.content}"
|
||||
)
|
||||
@@ -1829,6 +1868,7 @@ def _execute_tool(
|
||||
return (
|
||||
f"Document {doc.source}/{doc.path}:\n"
|
||||
f"date: {doc.created_at:%Y-%m-%d}\n"
|
||||
f"{_origin_lines(origin)}"
|
||||
f"lines: {first}-{end} of {total_lines}\n"
|
||||
f"{marker}{slice_text[:cap]}\n"
|
||||
f"{TRUNCATION_MARKER}\n"
|
||||
@@ -1842,6 +1882,7 @@ def _execute_tool(
|
||||
return (
|
||||
f"Document {doc.source}/{doc.path}:\n"
|
||||
f"date: {doc.created_at:%Y-%m-%d}\n"
|
||||
f"{_origin_lines(origin)}"
|
||||
f"lines: {first}-{end} of {total_lines}\n"
|
||||
f"{marker}{slice_text}"
|
||||
)
|
||||
|
||||
@@ -86,6 +86,83 @@ def sanitize_url(url: str) -> str:
|
||||
return _USERINFO_RE.sub(r"\1", url, count=1)
|
||||
|
||||
|
||||
def _url_basename(url: str) -> str:
|
||||
"""Phase 145 (task 01) — the repository directory a git row's URL
|
||||
imports into.
|
||||
|
||||
The URL's path component — after the ``://``, or after the ``:``
|
||||
for scp-style ``git@host:owner/repo`` (the port-carrying
|
||||
``ssh://user@host:22/…`` form already fell into the ``://``
|
||||
branch) — with trailing ``/`` stripped, a single trailing ``.git``
|
||||
suffix stripped, and the last segment taken. The importer clones
|
||||
into (and names the source after) exactly this basename
|
||||
(``source = root.name``), so it is the join key between a
|
||||
document's ``source`` name and the ``git_sources`` row whose clone
|
||||
produced it. An ``@`` inside the *path* is not userinfo and is
|
||||
never treated as one (no split, no strip).
|
||||
"""
|
||||
rest = url
|
||||
if "://" in rest:
|
||||
rest = rest.partition("://")[2]
|
||||
elif ":" in rest:
|
||||
# scp-style ``git@host:owner/repo`` — the path component starts
|
||||
# after the colon (``host:port`` has no other form here: the
|
||||
# only colon-carrying scheme without ``//`` is the scp one).
|
||||
rest = rest.partition(":")[2]
|
||||
rest = rest.rstrip("/")
|
||||
if rest.endswith(".git"):
|
||||
rest = rest[: -len(".git")]
|
||||
return rest.rsplit("/", 1)[-1]
|
||||
|
||||
|
||||
def origin_for_source(db: Session, source_name: str) -> tuple[str, str | None] | None:
|
||||
"""Phase 145 (task 01) — the origin a read result of *source_name*
|
||||
documents carries: the sanitized repository URL (and the pinned
|
||||
branch, when one exists), or ``None``.
|
||||
|
||||
The candidate rows are the ``kind='git'`` rows (local rows are
|
||||
never origins — nothing to link to from the user's browser); a row
|
||||
matches when its URL basename (:func:`_url_basename` — the clone
|
||||
directory the importer names the source after) equals
|
||||
``source_name``.
|
||||
|
||||
* EXACTLY ONE match → ``(sanitize_url(row.url), row.branch)`` —
|
||||
the phase-121 output mask keeps a legacy embedded-credential
|
||||
row's secret out of the prompt (the stored value is never
|
||||
modified — the :func:`clone_url_for` lesson), and ``branch`` is
|
||||
the pinned branch or ``None`` (NULL = the default branch);
|
||||
* zero matches, or TWO-OR-MORE (two repos sharing a basename) →
|
||||
``None`` + a WARNING log (the ambiguity names the sanitized
|
||||
candidates) — never guess: the read result would otherwise
|
||||
carry a wrong origin into every answer about the document, and
|
||||
a ``None`` result is byte-identical to pre-phase.
|
||||
|
||||
Pure-DB helper — one query, no git subprocess
|
||||
(``scripts/git_sync.py`` stays the only git-invocation site, PLAN
|
||||
§13), no caching (the table is a handful of rows; a read's extra
|
||||
query is trivial).
|
||||
"""
|
||||
rows = db.scalars(select(GitSource).where(GitSource.kind == "git")).all()
|
||||
candidates = [row for row in rows if _url_basename(row.url) == source_name]
|
||||
if len(candidates) == 1:
|
||||
row = candidates[0]
|
||||
return sanitize_url(row.url), row.branch
|
||||
if not candidates:
|
||||
logger.warning(
|
||||
"no git origin for source %r: no kind='git' row's URL basename matches",
|
||||
source_name,
|
||||
)
|
||||
return None
|
||||
logger.warning(
|
||||
"ambiguous git origin for source %r: %d rows share the URL basename (%s) — "
|
||||
"no origin line is added to its read results",
|
||||
source_name,
|
||||
len(candidates),
|
||||
", ".join(sanitize_url(row.url) for row in candidates),
|
||||
)
|
||||
return None
|
||||
|
||||
|
||||
def clone_url_for(row: GitSource) -> str:
|
||||
"""Phase 121, LOCKED A2 — the URL git actually clones, with the
|
||||
row's credential injected ONLY here.
|
||||
|
||||
+45
-1
@@ -231,6 +231,40 @@ SUGGEST_INTRO = (
|
||||
#: summaries are the starting points, not the content; ``grep`` with
|
||||
#: only ``pattern`` searches the whole knowledge base, and a source
|
||||
#: name is not a document.
|
||||
#: Phase 145 (task 02) — the original-source rule the ``read`` teaching
|
||||
#: carries (LOCKED verbatim — pinned in :mod:`tests.unit.test_prompt_lock`
|
||||
#: and :mod:`tests.unit.test_prompts`). :data:`TOOLS_SECTION` embeds this
|
||||
#: exact sentence-group at the ``read`` clause's end (the truncation
|
||||
#: workflow sentence) and BEFORE the ``grep`` clause begins; the
|
||||
#: cross-pins (``TOOLS_SECTION`` minus this rule reproduces the
|
||||
#: pre-phase-145 bytes; ``count == 1`` + the boundary substring) keep
|
||||
#: the embedded copy and this constant byte-identical. A read result for
|
||||
#: a git-sourced document carries an ``origin:`` line (and a ``branch:``
|
||||
#: line when a branch is pinned) — the read contract's fact — so when
|
||||
#: the answer uses such a document it includes the original source: a
|
||||
#: link to the repository (the ``origin`` line) with the document's path
|
||||
#: (the ``Document …`` header) named and the pinned branch when one is
|
||||
#: shown, as the repository's file-view URL where the model can tell it
|
||||
#: (``<repo-url>/blob/<branch>/<path>``), or the repository URL plus the
|
||||
#: path as plain text otherwise — the user opens the complete original
|
||||
#: from the answer in one step, not a second question. The rule is
|
||||
#: CONDITIONAL on the ``origin`` line's presence — a local-sourced
|
||||
#: document's read result carries none, so the rule adds nothing to
|
||||
#: those answers (the model has no URL to cite; it must not invent one —
|
||||
#: the sentence's conditioning does that job).
|
||||
ORIGIN_RULE: str = (
|
||||
"A read result for a document from a git source carries an origin line "
|
||||
"(and a branch line when a branch is pinned): when your answer uses "
|
||||
"such a document, include the original source in the answer — a link "
|
||||
"to the repository from the origin line, with the document's path "
|
||||
"(from the Document header) named and the pinned branch when one is "
|
||||
"shown — as the repository's file-view URL where you can tell it "
|
||||
"(for example <repo-url>/blob/<branch>/<path>), or the repository "
|
||||
"URL plus the path as plain text otherwise. The user should be able "
|
||||
"to open the complete original from your answer in one step, not ask "
|
||||
"again for the source."
|
||||
)
|
||||
|
||||
TOOLS_SECTION: str = (
|
||||
"<tools>\n"
|
||||
"You may extend your context with three tools. `ls` lists the "
|
||||
@@ -260,7 +294,17 @@ TOOLS_SECTION: str = (
|
||||
"are capped: a "
|
||||
"cut read returns the first part plus a TRUNCATED notice — the "
|
||||
"document did not end where it stopped; use `grep` (pattern) to "
|
||||
"find the rest, it searches the whole document. `grep` locates an exact string "
|
||||
"find the rest, it searches the whole document. A read result for a "
|
||||
"document from a git source carries an origin line (and a branch line "
|
||||
"when a branch is pinned): when your answer uses such a document, "
|
||||
"include the original source in the answer — a link to the "
|
||||
"repository from the origin line, with the document's path (from the "
|
||||
"Document header) named and the pinned branch when one is shown — as "
|
||||
"the repository's file-view URL where you can tell it (for example "
|
||||
"<repo-url>/blob/<branch>/<path>), or the repository URL plus the "
|
||||
"path as plain text otherwise. The user should be able to open the "
|
||||
"complete original from your answer in one step, not ask again for "
|
||||
"the source. `grep` locates an exact string "
|
||||
"(case-insensitive) in the indexed documents and returns up to 20 "
|
||||
"matching `source/path:line: text` lines — a locator, not a "
|
||||
"context-adder: read the winner with `read`. A grep pattern is a "
|
||||
|
||||
Reference in New Issue
Block a user