phase: 106_document_dates
Everything is verified green. Final report: **Phase 106 — Document dates (final verification pass; all 10 tasks already complete)** - Verified all phase artifacts: alembic `0020` (dev DB at `0020`), `doc_dates.py`, git `file_commit_dates`, importer `doc_dates_by_root`/`dates_updated`, both entry-point wirings, date APIs + tree `created_at`/`updated_at`, LLM surfaces (prompt block, `read` line 2, appended `ls` field), `apply_recency_boost` in `retrieve()`, UI columns/badge, admin editor, mock-LLM regex — all present and correct; no defects found, no fixes needed. - `uv run pytest --cov=app --cov-report=term-missing` → **2299 passed, TOTAL 99%** (>90% ✓) - `uv run pytest tests/e2e/test_document_dates.py -v --no-cov` → **6/6 passed** in isolation (DB up) - 12 regression E2E suites (retrieval_quality, whole_document_context, agent_document_tools, ls_tree_drilldown, read_truncation_cap, kb_tree, kb_tree_nav, document_viewer, edit_summaries, import_documents, sync_button, hidden_folders_toggle, smoke) → **all green in isolation** - `uv run ruff check .` → clean; `uv run pyright` → **0 errors, 0 warnings** **Completion criteria:** 1) non-null `created_at` + 0020 upgrade/downgrade on dev DB ✓ (real-Alembic integration tests) 2) sync refresh/older/manual-persists/content-reset/no sources_meta bump ✓ 3) zip/tar mtime + future→today ✓ 4) LLM date surfaces + cross-check ✓ 5) UI Created/Updated/badge positions ✓ 6) admin editor set+revert round-trip ✓ 7) old-correct-beats-new-similar (defaults & boost-off) + near-tie + `BOR_RECENCY_BOOST=0` byte-identical ✓ 8) full gate ✓ 9) commit/phase-move — left to harness per instructions. - **Notable:** recency default tuned 0.001 → **0.0007** (task 07 step 5 explicitly permits; measured margins recorded in `test_recency_boost.py` docstring). - **Next pending phase:** none — `todo/` holds only this phase.
This commit is contained in:
+60
-36
@@ -70,9 +70,11 @@ task 04):
|
||||
(the count is the subfolder's recursive subtree — every document
|
||||
whose path equals the folder or starts with ``folder + "/"``, the
|
||||
same set the sync-time folder summary describes — and the file
|
||||
lines ``source: X | path: Y | title: Z`` (the canonical
|
||||
``read``/``grep`` identity — the phase-63 labeled format,
|
||||
unchanged) in path order (``GET /api/docs`` order), capped at
|
||||
lines ``source: X | path: Y | title: Z | date: YYYY-MM-DD`` (the
|
||||
canonical ``read``/``grep`` identity — the phase-63 labeled format —
|
||||
plus the phase-106 D5 ``date`` field APPENDED after ``title``; only
|
||||
FILE lines carry a date — source/folder lines are not documents)
|
||||
in path order (``GET /api/docs`` order), capped at
|
||||
:data:`LS_MAX_FILE_LINES` lines + one deterministic grep-pointer
|
||||
note for the rest (a 500-file folder costs 50 lines, never 500);
|
||||
a ``source/folder`` ``path``: that folder's subfolders + own file
|
||||
@@ -463,7 +465,8 @@ NO_DOCUMENT_DID_YOU_MEAN_MANY = (
|
||||
SUGGESTION_LIMIT = 3
|
||||
|
||||
#: The drill-down ``ls`` file-line cap (phase 94, task 03): a folder's
|
||||
#: own files list at most this many ``source: X | path: Y | title: Z``
|
||||
#: own files list at most this many
|
||||
#: ``source: X | path: Y | title: Z | date: YYYY-MM-DD``
|
||||
#: lines (path order), then one deterministic grep-pointer note — a
|
||||
#: 500-file folder costs the model 50 lines + the note, never 500.
|
||||
#: Pinned module constant (no env var — the phase-94 TODO asks for a
|
||||
@@ -655,14 +658,17 @@ def _source_root_summaries(db: Session) -> list[tuple[str, str]]:
|
||||
]
|
||||
|
||||
|
||||
def _source_document_rows(db: Session, source: str) -> list[tuple[str, str]]:
|
||||
"""``(path, title)`` of every document under *source*, ordered by
|
||||
``path`` — the one bounded fetch a folder drill level lists (phase
|
||||
94 task 03; one source's paths, not the whole KB)."""
|
||||
def _source_document_rows(db: Session, source: str) -> list[tuple[str, str, str]]:
|
||||
"""``(path, title, created_iso_date)`` of every document under
|
||||
*source*, ordered by ``path`` — the one bounded fetch a folder
|
||||
drill level lists (phase 94 task 03; one source's paths, not the
|
||||
whole KB). The date is the row's ``created_at`` UTC date part
|
||||
(``YYYY-MM-DD``, phase 106 D5 — the ``ls`` FILE line's appended
|
||||
`` | date: …`` field; only file lines carry a date)."""
|
||||
return [
|
||||
(path, title)
|
||||
for path, title in db.execute(
|
||||
select(Document.path, Document.title)
|
||||
(path, title, created_at.strftime("%Y-%m-%d"))
|
||||
for path, title, created_at in db.execute(
|
||||
select(Document.path, Document.title, Document.created_at)
|
||||
.where(Document.source == source)
|
||||
.order_by(Document.path)
|
||||
)
|
||||
@@ -707,15 +713,17 @@ def ls_top(db: Session) -> list[tuple[str, int, str | None]]:
|
||||
def group_folder_listing(
|
||||
source: str,
|
||||
folder: str,
|
||||
rows: Sequence[tuple[str, str]],
|
||||
rows: Sequence[tuple[str, str, str]],
|
||||
summaries: Mapping[str, str],
|
||||
) -> tuple[list[tuple[str, int, str | None]], list[tuple[str, str, str]], int]:
|
||||
) -> tuple[
|
||||
list[tuple[str, int, str | None]], list[tuple[str, str, str, str]], int
|
||||
]:
|
||||
"""One level of the drill-down tree (phase 94, task 03) — pure.
|
||||
|
||||
Given *rows* — the source's ``(path, title)`` pairs in catalog
|
||||
(path) order — and *summaries* (the source's stored
|
||||
``folder_summaries`` rows: ``folder_path → summary``), the folder
|
||||
level *folder* (source-relative; ``""`` = the source root):
|
||||
Given *rows* — the source's ``(path, title, created_iso_date)``
|
||||
triples in catalog (path) order — and *summaries* (the source's
|
||||
stored ``folder_summaries`` rows: ``folder_path → summary``), the
|
||||
folder level *folder* (source-relative; ``""`` = the source root):
|
||||
|
||||
* **(a) direct subfolders** — the folders whose parent is exactly
|
||||
*folder*, in path order, each
|
||||
@@ -732,8 +740,10 @@ def group_folder_listing(
|
||||
prefix before the last ``/`` —
|
||||
:func:`app.rag.folder_summaries.folder_of`, the shared notion) IS
|
||||
*folder*, in path order (catalog order — the same order
|
||||
``GET /api/docs`` serves), as ``(source, path, title)`` triples
|
||||
— the canonical ``read``/``grep`` identity, capped at
|
||||
``GET /api/docs`` serves), as ``(source, path, title, date)``
|
||||
4-tuples — the canonical ``read``/``grep`` identity plus the
|
||||
phase-106 D5 ``date`` field (the row's ``created_at`` UTC date
|
||||
part, APPENDED — never inserted before ``title``), capped at
|
||||
:data:`LS_MAX_FILE_LINES` (the rest fold into the renderer's
|
||||
note; a 500-file folder never costs 500 lines).
|
||||
* **(c) the TOTAL direct-file count** — pre-cap, for the note.
|
||||
@@ -745,7 +755,7 @@ def group_folder_listing(
|
||||
# indexed path (the existence rule's candidate set — a folder is
|
||||
# present iff at least one path starts with ``folder + "/"``).
|
||||
folders: set[str] = set()
|
||||
for path, _title in rows:
|
||||
for path, _title, _date in rows:
|
||||
f = folder_of(path)
|
||||
while f:
|
||||
folders.add(f)
|
||||
@@ -755,7 +765,7 @@ def group_folder_listing(
|
||||
# folder + "/"`` arm (the folder's true descendants), one pass per
|
||||
# document.
|
||||
counts: dict[str, int] = {f: 0 for f in folders}
|
||||
for path, _title in rows:
|
||||
for path, _title, _date in rows:
|
||||
if path in folders:
|
||||
counts[path] += 1
|
||||
f = folder_of(path)
|
||||
@@ -767,8 +777,8 @@ def group_folder_listing(
|
||||
for g in sorted(g for g in folders if folder_of(g) == folder)
|
||||
]
|
||||
files = [
|
||||
(source, path, title)
|
||||
for path, title in rows
|
||||
(source, path, title, date)
|
||||
for path, title, date in rows
|
||||
if folder_of(path) == folder
|
||||
]
|
||||
return subfolders, files[:LS_MAX_FILE_LINES], len(files)
|
||||
@@ -776,7 +786,9 @@ def group_folder_listing(
|
||||
|
||||
def ls_folder(
|
||||
db: Session, source: str, folder: str
|
||||
) -> tuple[list[tuple[str, int, str | None]], list[tuple[str, str, str]], int]:
|
||||
) -> tuple[
|
||||
list[tuple[str, int, str | None]], list[tuple[str, str, str, str]], int
|
||||
]:
|
||||
"""One folder level of the drill-down ``ls`` (phase 94, task 03).
|
||||
|
||||
The source's document rows (:func:`_source_document_rows`) and
|
||||
@@ -793,7 +805,7 @@ def ls_folder(
|
||||
)
|
||||
|
||||
|
||||
def _folder_exists_in(rows: Sequence[tuple[str, str]], folder: str) -> bool:
|
||||
def _folder_exists_in(rows: Sequence[tuple[str, str, str]], folder: str) -> bool:
|
||||
"""The phase-94 folder-existence rule (``00_phase.md``), pure.
|
||||
|
||||
Folder *folder* (source-relative) under a registered source
|
||||
@@ -805,11 +817,11 @@ def _folder_exists_in(rows: Sequence[tuple[str, str]], folder: str) -> bool:
|
||||
if not folder:
|
||||
return True
|
||||
prefix = folder + "/"
|
||||
return any(path.startswith(prefix) for path, _title in rows)
|
||||
return any(path.startswith(prefix) for path, _title, _date in rows)
|
||||
|
||||
|
||||
def _deepest_existing_ancestor(
|
||||
rows: Sequence[tuple[str, str]], folder: str
|
||||
rows: Sequence[tuple[str, str, str]], folder: str
|
||||
) -> str:
|
||||
"""The deepest EXISTING folder prefix of a missing *folder* (pure).
|
||||
|
||||
@@ -854,7 +866,7 @@ def render_ls_top(entries: Sequence[tuple[str, int, str | None]]) -> str:
|
||||
def render_folder_listing(
|
||||
identity: str,
|
||||
subfolders: Sequence[tuple[str, int, str | None]],
|
||||
files: Sequence[tuple[str, str, str]],
|
||||
files: Sequence[tuple[str, str, str, str]],
|
||||
total_files: int,
|
||||
) -> str:
|
||||
"""One folder level of the drill-down ``ls`` (phase 94, task 03) —
|
||||
@@ -867,9 +879,10 @@ def render_folder_listing(
|
||||
below the header — a blank line, the 2-space-indented subfolder
|
||||
lines `` {sub}/ — {m} documents`` in path order (``: {summary}``
|
||||
appended ONLY when the subfolder's summary is stored), a blank
|
||||
line, the file lines in EXACTLY the existing
|
||||
``source: X | path: Y | title: Z`` format (the canonical
|
||||
``read``/``grep`` identity — unchanged), and the cap note
|
||||
line, the file lines in EXACTLY the
|
||||
``source: X | path: Y | title: Z | date: YYYY-MM-DD`` format (the
|
||||
canonical ``read``/``grep`` identity plus the phase-106 D5
|
||||
appended ``date`` field — the only changed part), and the cap note
|
||||
``…and {hidden} more documents in this folder — use grep
|
||||
(pattern) to find a specific one.`` ONLY when the folder's own
|
||||
files outnumber :data:`LS_MAX_FILE_LINES` (*files* arrives capped;
|
||||
@@ -891,8 +904,8 @@ def render_folder_listing(
|
||||
if files or total_files > len(files):
|
||||
body.append("")
|
||||
body.extend(
|
||||
f"source: {source} | path: {path} | title: {title}"
|
||||
for source, path, title in files
|
||||
f"source: {source} | path: {path} | title: {title} | date: {date}"
|
||||
for source, path, title, date in files
|
||||
)
|
||||
hidden = total_files - len(files)
|
||||
if hidden > 0:
|
||||
@@ -1172,15 +1185,26 @@ def _execute_tool(
|
||||
holder.read_truncations.append(
|
||||
(cast("str", raw_path), cap, len(doc.content))
|
||||
)
|
||||
# Phase 106 (D5): the date rides every document the model
|
||||
# sees — the ``read`` result's SECOND line; the FIRST line
|
||||
# stays ``Document {source}/{path}:`` BYTE-IDENTICAL (the
|
||||
# E2E mock's ``_READ_RESULT_PREFIX`` header contract).
|
||||
return (
|
||||
f"Document {doc.source}/{doc.path}:\n"
|
||||
f"date: {doc.created_at:%Y-%m-%d}\n"
|
||||
f"{doc.content[:cap]}\n"
|
||||
f"{TRUNCATION_MARKER}\n"
|
||||
f"{READ_TRUNCATION_NOTICE.format(shown=cap, total=len(doc.content))}"
|
||||
)
|
||||
# At or under the cap: byte-identical to the pre-phase-95 result
|
||||
# (no marker, no notice, no holder entry, no ToolResultPiece).
|
||||
return f"Document {doc.source}/{doc.path}:\n{doc.content}"
|
||||
# At or under the cap: the pre-phase-95 result plus the
|
||||
# phase-106 D5 date line (first line byte-identical — the
|
||||
# mock's header contract; no marker, no notice, no holder
|
||||
# entry, no ToolResultPiece).
|
||||
return (
|
||||
f"Document {doc.source}/{doc.path}:\n"
|
||||
f"date: {doc.created_at:%Y-%m-%d}\n"
|
||||
f"{doc.content}"
|
||||
)
|
||||
if call.name == "grep":
|
||||
raw_pattern = call.arguments.get("pattern")
|
||||
pattern = raw_pattern.strip() if isinstance(raw_pattern, str) else ""
|
||||
|
||||
Reference in New Issue
Block a user