diff --git a/.agents/phases/todo/106_document_dates/00_phase.md b/.agents/phases/todo/106_document_dates/00_phase.md new file mode 100644 index 0000000..d9d37ba --- /dev/null +++ b/.agents/phases/todo/106_document_dates/00_phase.md @@ -0,0 +1,106 @@ +# Phase 106 — Document dates end to end: sourced at sync, displayed, editable, recency-weighted in retrieval + +**Source:** Owner request 2026-09-13 (chat): documents "all have dates, either via their git timestamp or via file metadata (hopefully) preserved in the tar or zip archive process"; "After this phase, all documents must include the date when fed to the LLM"; "The UI must also show a date for every document at the top of that document when the user clicks it"; "If the date of a document can't be determined or is in the future then assume that document was created today"; "Date should be stored in the bor database and updated when sources are synced"; "Newer documents should be ranked higher in retrieval somehow, or at least given a boost, without breaking the existing retrieval process"; "last updated dates/timestamps on folders before the description column but after the documents column"; "For files, include a date/timestamp before the 'indexed' column"; "This timestamp should be editable so users can correct for errors"; "the timestamp can't be null so just set it to today's date during the migration and then, on sync, update the date"; "It's totally fine for a sync to cause a document or folder's date to get older." + +**Story:** n/a (owner roadmap confirmation — the phase's E2E suite proves the item end to end). + +**Context:** Every imported file becomes a `documents` row (`app/models.py` L101) with `indexed_at` (L112, the INDEX time — a different concept from this phase's document *creation* date, which it does not touch) and a nullable `summary`. The importer (`app/rag/importer.py`) walks each source root (`iter_importable_files` L171-194) and upserts per file in `_index_file` (L376-471): content-hash delta — `added`/`updated` (the row write, L405-421) or the early-return `unchanged` path (L401-404, **no write today**); `import_sources` (L213-340) already carries per-root maps keyed by `str(root)` — `ignore_by_root` (phase 89) and `include_hidden_by_root` (phase 105) — with the progress pre-walk and prune using the same per-root resolution; `ImportSummary` (L64-111) logs the greppable `import: summary files=…` line (PLAN §9). The two live entry points both resolve the `git_sources` rows, clone git rows through `scripts/git_sync.clone_or_pull` (`app/api/sync.py::_run_sync` per-row loop L233-263, import call L300-303; `scripts/import_docs.py::_resolve_sources` L168-226, clone L220, import call L331-334) and pass the per-root maps through — this phase adds a THIRD map (task 04). **Verified git behavior (2026-09-13, scratch repos):** `git clone --depth 1 /local/path` warns "--depth is ignored in local clones" and keeps FULL history, so local-path git sources can yield TRUE per-file last-commit dates; URL-transport clones (https/ssh/`file://`) are shallow, and in a shallow clone `git log -1 --format=%cI -- ` returns the TIP commit's date for EVERY existing file (the shallow boundary is each file's history root — a uniform per-repo date, real across repos, flat within one). `git log --name-only --format=@@%cI` walks newest-first and lists every working-tree file under the shallow tip. The archive unpacker (`app/rag/archive_upload.py`, phase 49) streams member content (`_unpack_zip` L214-238, `_unpack_tar` L241-270) but **never restores member mtimes** — extracted files carry the extraction time, so uploads today lose their dates (the owner's "hopefully" — this phase fixes it, task 03). Retrieval (A7): cosine top-100 ∪ FTS top-30 → RRF `fuse()` (k=60, `app/rag/retriever.py` L225-258) → `select_documents` (L540+, sorts chunks by fused `score`, dedupes parents, top-N=2 full texts, never truncated); the honesty gate and `query_log.top_score` are COSINE-based (untouched by a score-side boost); lexical candidates are detached `Document` rows rebuilt from two raw SQL projections (`_LEXICAL_SQL` L140-157, `_NAME_HIT_SQL` L277-300) that must gain the new column (task 06); the vector path returns ORM rows (the column comes for free). LLM surfaces: the HIGH prompt's `` block (the pinned shape, `app/rag/prompts.py` L371-376: `source`/`path`/`title` attributes + full text), the `read` tool result (`app/rag/agent.py` L1173-1185: first line `Document {source}/{path}:` — the E2E mock's `_READ_RESULT_PREFIX` header contract, `tests/e2e/mock_llm.py` L867), and the `ls` file line (`render_folder_listing` L856-905: `source: X | path: Y | title: Z`, fed by `_source_document_rows` L658-669 + the pure `group_folder_listing` L707-780, cross-checked level-for-level against the UI tree builder in `app/api/docs.py` L326/L376). The E2E mock parses prompts/tool results with regexes that must stay compatible: `_DOCUMENT_BLOCK_RE` (L785-789 — breaks unless updated for the new attribute), `_CATALOG_LINE_RE` (L873-875 — `title: .+$` tolerates an APPENDED ` | date: …` field), `_READ_RESULT_PREFIX` (L867 — header line must stay byte-identical). UI: the RAG view's file table header `Source | Path | Title | Chunks | Indexed` (`frontend/index.html` L485-492, rows built by `makeRow` in `frontend/assets/sources.js` L1375-1410 — cell order `[title, chunks, indexed_at]` L1401) and the folder/source table `Folder | Documents | Description` (L470-478, rows `makeSourceRow` L1217-1235 / `makeFolderRow` L1237-1261); the view drills client-side over `GET /api/docs/tree` (phase 97/99 — pure builder `build_kb_tree` `app/api/docs.py` L377-490 over `TreeDocRow` tuples L296-302; file nodes carry `indexed_at` verbatim, folder nodes carry recursive `documents` counts); the document viewer (click → same-page modal, phase 26; full-page escape `/document.html`) renders through ONE shared core `renderDocument` (`frontend/assets/document.js` L118-176: title + the `.doc-meta` badge row — `source · format · path · Indexed · chunks` — then content; `fmtDate` L85, `metaBadge` L97); the admin-only inline editor idiom is phase 57's summary editor (`docAdminReady()` gate, `wireSummaryEdit`, `PATCH /api/documents/summary` in `app/api/docs.py` L164-232 — embed-before-write, 404 unknown pair, admin gate). House patterns: Alembic head is `0019` (every migration ships a tested downgrade, A13); env settings are `BOR_`-prefixed `Settings` fields with the kill-switch validator pattern (`agent_max_rounds`: 0 disables, negative fails startup loudly, `app/config.py` L155-168); the `manually_edited` flag (`folder_summaries`, phase 97) is the precedent for owner corrections surviving the sync-time generator; integration tests seed real Postgres with deterministic axis vectors (`tests/integration/test_name_hit_lexical.py` — `D=768` unit vectors, exact cosines); E2E story suites seed through the REAL importer against a fixture dir with the deterministic mock LLM (`tests/e2e/test_retrieval_quality.py` — `_import_fixtures`/`_reset_db`/source-chip assertions); `tests/conftest.py` holds the `db` fixture; `tests/fakes.py` the fake LLM. + +## Objective +Every indexed document carries a **creation date** from its source — the git last-commit date for git sources (true per-file for local-path checkouts, the repo tip-commit date for shallow URL checkouts), the file mtime for local directories and unpacked uploads (the unpackers now preserve member mtimes) — normalized (undetermined or future → today), stored NOT-NULL on `documents.created_at` (existing rows backfilled to the migration day per the owner), refreshed on every sync (allowed to go older), and editable by the admin (the correction survives syncs via a manual flag). The date is fed to the LLM on every document surface (the HIGH prompt's `` block, the `read` result, the `ls` file line), shown in the UI at the top of a clicked document (viewer badge) and in both catalog tables (files: `Created` before `Indexed`; folders/sources: `Updated` after `Documents`, before `Description` — the subtree's max document date), and applied in retrieval as a small, env-tunable, kill-switchable recency boost on the RRF-fused score that NEVER lets a newer similar document outrank an older document that actually answers the question (pinned by a dedicated integration battery + the story E2E). + +## Dependencies +- `105_hidden_folders_toggle` (complete) — the per-root `str(root)` map idiom in the importer both entry points already build (task 04 adds the third map alongside); its suites must stay green. +- `09_story_retrieval_quality` (complete) — the A7 hybrid retriever, `fuse()`/`select_documents`, and the eval script the boost extends; the fixture-KB E2E pattern. +- `97_kb_tree_catalog` / `99_kb_tree_table_and_back_nav` (complete) — the pure tree builder + the RAG view tables/rows this phase's columns extend; the ls↔tree cross-check that a changed `ls` line must keep. +- `57_edit_document_summaries` (complete) — the admin-only inline editor + `PATCH /api/documents/…` idiom the date editor copies. +- `28_git_based_sources` (complete) — `scripts/git_sync.py` (the ONLY git-invocation site, A11) where the per-file date walk is added; `--depth 1` stays (D10). +- `90_upload_no_scan` / phase 49 uploads (complete) — the unpacker this phase makes mtime-preserving (task 03); the scan-deferred upload flow itself is untouched. + +## Design (shared by all tasks — the executor reads this, not the chat) + +- **Storage (task 01, D1).** `documents.created_at` — `DateTime(timezone=True)` **NOT NULL**, `server_default=func.now()` — so every row of an existing deployment backfills to the migration moment (≈ today, the owner's instruction) and the next sync replaces it with the real sourced date (D4 makes the unchanged path write). `documents.created_at_manual` — `Boolean` NOT NULL, Python `default=False`, `server_default=text("false")` (the `include_hidden` L159-167 / `manually_edited` phase-97 column style). Alembic `0020_documents_created_at.py` revises `0019`; downgrade drops BOTH columns (A13). +- **Normalization (task 02, D3).** New module `app/rag/doc_dates.py` — pure, stdlib-only, the single choke point every date passes through: + ```python + FUTURE_SKEW_TOLERANCE = timedelta(days=1) + def normalize_doc_date(raw: datetime | None, now: datetime | None = None) -> datetime + ``` + `None` (undetermined) → `now` (default `datetime.now(UTC)`, date-part-irrelevant — full precision kept); naive `raw` → treated as UTC; aware `raw` → converted to UTC; `raw > now + FUTURE_SKEW_TOLERANCE` (a genuinely FUTURE date — the 1-day tolerance absorbs clock skew, pinned) → `now`. Also `file_mtime_datetime(path: Path) -> datetime` = `datetime.fromtimestamp(path.stat().st_mtime, tz=UTC)` (epoch mtimes are tz-agnostic — UTC is the correct rendering). No other module re-derives these rules. +- **Source date extraction (task 03, D2/D10).** + - **Archives** — `_unpack_zip`: after each regular-file write, `os.utime(dest, ns=(t, t))` from the member's DOS `date_time` (`datetime(*member.date_time, tzinfo=timezone.utc).timestamp()` — DOS mtimes are tz-agnostic like epochs); `_unpack_tar`: `os.utime(dest, ns=(member.mtime, member.mtime))`. Regular files ONLY (dirs/symlinks/hardlinks untouched — only regular files are ever indexed); the zip-bomb cap and every safety check are UNCHANGED (the utime calls come after `_write_capped` succeeds — a failed unpack still removes the partial tree). + - **Git** — `scripts/git_sync.py` gains `file_commit_dates(dest: Path) -> dict[str, datetime]`: ONE `git log --name-only --format=@@%cI` walk through `run_git` (the A11 single-invocation site), parsed newest-first — per repo-relative POSIX path, the FIRST sighting wins (the latest commit that touched it); paths normalized to POSIX (`/`); ISO-strict parsed via `datetime.fromisoformat`. **Fail-soft:** any `GitSyncError`/parse failure → `{}` (logged) — the importer falls back to file mtimes; a date walk must never break a sync. Verified behavior: local-path checkouts (full history) → true per-file dates; shallow URL checkouts → the tip commit's date for every working-tree file (the shallow-boundary property, pinned by an integration test using a `file://` clone). +- **Importer semantics (task 04, D4).** `import_sources(…, doc_dates_by_root: dict[str, dict[str, datetime]] | None = None)` — keyed by `str(root)` exactly like `ignore_by_root`/`include_hidden_by_root`; ONLY git roots are listed (the entry points build the map from `file_commit_dates` after the clone); unlisted roots (local dirs, uploads) take the mtime fallback. In `_index_file` (receives `raw_date: datetime | None` pre-resolved by the loop — `map.get(rel)` else `None`): + - `raw_date is None` → `file_mtime_datetime(full_path)` (one `stat`). + - **added / updated (content change)** → `doc.created_at = normalize_doc_date(raw)`, `doc.created_at_manual = False` — a new content version is a new document date; a previous manual correction referred to the old content and is deliberately reset. + - **unchanged (hash match)** → if `doc.created_at_manual` → leave the row entirely (the owner's correction survives syncs — the phase-97 `manually_edited` precedent); else recompute `normalize_doc_date(raw)` and, when it differs from the stored value, write + commit it (a date-only refresh) and count it in the NEW `ImportSummary.dates_updated` (additive field + the `import: summary … dates_updated=N` log term, PLAN §9). A date-only refresh is still `unchanged`: it NEVER counts toward `added/updated/pruned` → no `sources_meta` bump, no overview/folder-summary regeneration, no saved-chat staleness (D4 — content is the KB, the date is annotation). + - Entry points: `_run_sync` — in the existing per-row loop, for `kind=git` rows after `clone_or_pull`: `doc_dates_by_root[str(root)] = file_commit_dates(root)`; pass the map to `import_sources`; add `"dates_updated": summary.dates_updated` to the success `detail` (additive key). `scripts/import_docs.py::_resolve_sources` returns the map as a 4th tuple element (manual `--source` dirs and env-fallback rows contribute nothing); `main` passes it through (L331-334). +- **API surface (task 05, D7/D8/D9).** + - `GET /api/docs` — `DocSummary` gains `created_at: str` (ISO-8601, `doc.created_at.isoformat()`) — the column joins the existing select + group_by (`indexed_at` L104/L107). `GET /api/documents/content` — `DocContent` gains `created_at: str` (L161 site). + - `PATCH /api/documents/date` (NEW, admin-only, `require_admin` — the `update_document_summary` gate L164-232): body `DateUpdate {source: str, path: str, date: str | None}`. Unknown pair → 404 `document not found` (row-lookup semantics, no filesystem — the `/documents/content` rule). `date` present → `datetime.fromisoformat` (a bare `YYYY-MM-DD` or full ISO datetime; a malformed value → the 422 the model gives) → `normalize_doc_date` (D3 — a manually set future date also folds to today; consistency with the sourced path) → `created_at` set + `created_at_manual = True`. `date` null/absent → the CLEAR: `created_at_manual = False` only (the stored date stands until the next sync refreshes it — the API is DB-only and cannot re-read the source). Response `DateResult {source, path, created_at: str, created_at_manual: bool}` echoing the stored state. No LLM/embedding call (a date is never embedded — the phase-57 no-LLM contrast). + - `GET /api/docs/tree` — `TreeDocRow` becomes a 6-tuple `(source, path, title, chunks, indexed_at, created_at_iso)`; the endpoint's query adds `Document.created_at` (the `indexed_at` select/group_by site L543-546); `KbTreeFile` gains `created_at: str`; `KbTreeFolder`/`KbTreeSource` gain `updated_at: str | None` (D9: the subtree's MAX `created_at`, computed in the PURE builder as it recurses — `None` for a 0-document source, the `summary`-null shape); the builder threads the max through `_source_node` → `_level_children` (folders: the max of own files + child subfolders). No new table, no new column beyond task 01's (D9). +- **LLM surfaces (task 06, D5).** + - Retriever plumbing: `_LEXICAL_SQL` + `_NAME_HIT_SQL` gain `d.created_at AS created_at`; the two detached `Document(…)` reconstructions (L360-372, L414-424) pass `created_at=row.created_at`. The vector path is ORM rows — nothing to do. + - HIGH prompt (`app/rag/prompts.py` L371-376): the block becomes `` — the UTC date part, attribute appended AFTER `title` (the only position; the attribute is always present — `created_at` is NOT NULL). No persona/teaching copy changes (phase 03 convention — the date rides existing locked text). + - `read` tool (`app/rag/agent.py` L1173-1185): the first line stays `Document {source}/{path}:` BYTE-IDENTICAL (the mock's `_READ_RESULT_PREFIX` header contract — `_read_results` strips exactly that header to recover the path), the date is the SECOND line: `Document {sp}:\ndate: {doc.created_at:%Y-%m-%d}\n{content…}` — both the truncated (L1176) and the plain (L1183) results. + - `ls` file line: APPEND ` | date: {YYYY-MM-DD}` at the END — `source: X | path: Y | title: Z | date: 2024-06-15` (the mock's `_CATALOG_LINE_RE` `title: .+$` still matches — the appended field lands inside the greedy tail; do NOT insert before `title`, where the non-greedy `path` capture would swallow it). Plumbing: `_source_document_rows` returns `(path, title, created_iso_date)`; `group_folder_listing` rows become `Sequence[tuple[str, str, str]]` and its file triples become `(source, path, title, date)`; `render_folder_listing` renders the appended field; the NOT-A-FOLDER refusal path (L1121-1130) passes through unchanged in shape; the `app/api/docs.py` ls↔tree cross-check docstrings + `tests/unit/test_kb_tree_builder.py` cross-check tests compare the extended tuples (the "UI shows what the agent sees" invariant, extended with the date). + - `tests/e2e/mock_llm.py` `_DOCUMENT_BLOCK_RE` (L785-789): make the date attribute an OPTIONAL group — `r'title="[^"]*"(\sdate="[^"]*")?>'` — so the mock tolerates pre- and post-phase shapes (house rule: marker/regex changes land with the prompt change, in this task). +- **Recency boost (task 07, D6).** `Settings` gains (the hybrid block, `app/config.py` L163-174): `recency_boost: float = 0.001` (the max additive score for a zero-age document; **`0` = off — pre-phase ranking byte-identical**, the kill switch; negative → startup failure naming the field, the `agent_max_rounds` validator pattern) and `recency_half_life_days: int = 365` (`<= 0` fails startup). `.env.example` documents `BOR_RECENCY_BOOST` / `BOR_RECENCY_HALF_LIFE_DAYS`. `app/rag/retriever.py` gains the pure `apply_recency_boost(chunks, *, now=None, weight=None, half_life_days=None) -> list[RetrievedChunk]`: for each chunk, `age_days = max(0.0, (now − doc.created_at).total_seconds() / 86400.0)` (future clamps to 0 — a future-sourced doc reads as brand-new, consistent with D3), `score += weight * math.exp(−age_days / half_life_days)` (defaults from `get_settings()` when args omitted), then re-sorted with the EXISTING deterministic key `(−score, −cosine, document.path, position)` — with `weight=0` the scores and order are untouched (pinned). `retrieve()` (L398-425) applies it AFTER `fuse()` when `settings.recency_boost > 0` (the single apply site — chat API and `scripts/eval_retrieval.py` get it automatically; the script's printed table gains the document date + the post-boost effective score column). UNTOUCHED by design: the A8 honesty gate + `query_log.top_score` (cosine), `weak_hit_titles` (titles only), the never-truncated top-N contract, `fuse()` itself. The boost re-ranks which ≤N documents are FED (the whole point) but never truncates. +- **The fine line (task 07's battery — the owner's warning, pinned permanently).** Real-Postgres integration tests with axis vectors (the `test_name_hit_lexical.py` idiom — exact cosines, deterministic): + 1. **Old-correct beats new-similar (THE owner scenario):** doc A (created 2020-01-01) answers the question; doc B (created yesterday) is topically similar (shares tokens, weaker answer). Under DEFAULTS A must be `select_documents()[0]`; ALSO under `recency_boost=0` (no-regression pin: relevance alone already ordered them). + 2. **Boost is real:** two near-twin documents (identical or 1-rank-apart fused scores, engineered so B's advantage is < the zero-age boost) — with defaults the NEWER ranks first; with `weight=0` the OLDER ranks first (proving the boost is the differentiator, not drift). + 3. **Decay:** the boost halves at the half-life (pure-function unit pins: age 0 → full `weight`; age = half-life → `weight/e`… asserted as `weight * exp(-1)`; age ≫ → ~0; future → full; `weight=0` → byte-identical list order). + 4. If the defaults (0.001 / 365 d) fail test 1's margin, tune the DEFAULTS (not the test) until old-correct wins with ≥ a comfortable fused-score margin, and record the measured margin in the test docstring — the owner re-tunes live via `BOR_RECENCY_BOOST`. +- **UI (tasks 08-09, D7/D8).** + - `frontend/index.html`: the file table header gains `Created` between `Chunks` and `Indexed` (L487-491); the folder table header gains `Updated` between `Documents` and `Description` (L473-475). No other shell change. + - `frontend/assets/sources.js`: `makeRow` — insert the `Created` cell BEFORE the `Indexed` one (the L1401 value list becomes `[title, chunks, fmtDate(d.created_at), fmtDate(d.indexed_at)]`; the row object fed from tree file nodes (L1341-1350) carries `created_at`); `makeSourceRow`/`makeFolderRow` — one `td` with `fmtDate(node.updated_at)` (or `"–"` when null — the `statLast` null idiom L1294) between the count `td` and the description cell. All `textContent` (the XSS contract — never innerHTML with document-derived data). + - `frontend/assets/document.js` (the ONE shared core — modal AND `/document.html` page): the `.doc-meta` badge row gains `metaBadge("doc-created", \`Created ${fmtDate(doc.created_at)}\`)` INSERTED BEFORE the `Indexed` badge (L127) — the date at the top of a clicked document (D8). The badge class reuses the `doc-indexed` styling family (a `doc-created` rule in `styles.css` next to it, provenance comment citing phase 106; contrast ≥4.5:1 verified + recorded in the comment, house style). + - Task 09 — the admin date editor (the phase-57 `wireSummaryEdit` idiom verbatim, gated by `docAdminReady()` — anonymous/token holders see the byte-identical phase-36/106 badge row, no button, no network call): an **`Edit date`** text button after the Created badge opens an inline editor in the meta row — a native `` (prefilled `doc.created_at`'s UTC date part, `aria-label` "Document creation date") + Save/Cancel buttons + a `role="status"` live line; Save → `PATCH /api/documents/date {source, path, date: }` — the endpoint is `require_admin` (the phase-57 split: the viewer content is `require_user`-gated, but the EDIT is admin-only; the editor is wired only for the admin, exactly as `wireSummaryEdit`); 200 → the badge re-renders from the response's `created_at`, `announcer` confirmation, status clears (§7.4); 4xx/5xx/network → the error line (`role="alert"`), the editor reverts to the stored value, controls re-enabled — the UI never claims a state the server didn't save. A **clear** affordance (the phase-57 "clear = empty" contrast): a "Revert to sync" link in the editor that sends `{source, path, date: null}` (D7's CLEAR — the manual flag drops; the date shows until the next sync). +- **NOT touched:** `indexed_at` (everywhere), the A7 fusion math / A8 gate / never-truncated contract, `sources_meta` staleness semantics, the clone strategy (`--depth 1`, D10), the upload scan-deferral (phase 90), suggestions, shared chats, the stat cards, `AGENTS.md`, `.agents/PLAN.md`, any completed phase. The `ls` top-level SOURCE lines and folder lines carry NO date (only FILE lines do — files are documents). + +## Tasks +1. `01_created_at_column.md` — `documents.created_at` + `documents.created_at_manual` (model + alembic `0020`) + default/round-trip/migration tests. +2. `02_date_normalization.md` — `app/rag/doc_dates.py` (`normalize_doc_date` + `file_mtime_datetime`) + boundary unit tests. +3. `03_source_date_extraction.md` — mtime-preserving zip/tar unpack + `scripts/git_sync.py::file_commit_dates` (local full-history vs shallow tip-date, fail-soft) + tests. +4. `04_importer_dates.md` — `import_sources` date map + the added/updated/unchanged semantics + `dates_updated` + both entry-point wirings + unit & integration tests. +5. `05_date_apis.md` — `created_at` on `/api/docs` + `/api/documents/content`, the admin `PATCH /api/documents/date`, and the tree's `created_at`/`updated_at` (schemas + pure builder + endpoint) + tests. +6. `06_llm_date_surfaces.md` — retriever `created_at` plumbing, the `` block, the `read` date line, the `ls` appended date field, the mock-LLM regex, pin updates + tests. +7. `07_recency_boost.md` — settings + validators, `apply_recency_boost` in `retrieve()`, `eval_retrieval` columns, `.env.example`, the fine-line integration battery + unit tests. +8. `08_ui_dates.md` — the `Created` file column, the `Updated` folder/source column, the viewer `Created` badge (+ CSS/a11y) + source-level unit pins. +9. `09_date_editor.md` — the admin-only inline date editor in the viewer (PATCH wiring, the clear/revert affordance, §7.4 lifecycle, a11y) + unit pins. +10. `10_e2e_document_dates.md` — dedicated Playwright suite `tests/e2e/test_document_dates.py` (isolation) + regressions + full gate + atomic commit. + +## Testing & Quality +- Unit — `tests/unit/test_doc_dates.py` (task 02, the boundary matrix: None→today; naive-as-UTC; aware conversion; future just inside the 1-day tolerance keeps its date; future beyond → today; the mtime helper); `tests/unit/test_archive_upload_dates.py` (task 03 — a zip with an explicit `date_time` and a tar with an explicit `mtime` unpack to files carrying those mtimes; the cap/safety suite stays green); the `file_commit_dates` pins (task 03 — a scratch repo: local clone → per-file dates; a `file://` shallow clone → tip date for every file; a missing `.git`/git failure → `{}` fail-soft); `tests/unit/test_importer_dates.py` (task 04 — a fake-LLM + tmp-tree: utime'd files land as `created_at`; the unchanged-refresh write + counter; the manual-flag skip; the content-change reset; `doc_dates_by_root` map hit vs mtime fallback); `tests/unit/test_kb_tree_builder.py` (extended, task 05 — file `created_at` verbatim, folder/source `updated_at` = subtree max, `None` for 0-doc sources, the extended cross-check tuples); `tests/unit/test_prompts_dates.py` (task 06 — the block attribute's position/format, the read line shape, the ls line's appended field, the weight-0 byte-identity of the boosted order lives in task 07's suite); `tests/unit/test_retriever_recency.py` (task 07 — the decay/weight/tie pins); `tests/unit/test_sources_dates.py` (tasks 08/09 — the house read-the-assets-as-text pattern: header cell ORDER pinned in both tables, the row cell order, the badge order in the shared core, the editor wiring + its aria labels + the single-source cross-file check that the editor PATCHes `/api/documents/date`). +- Integration — `tests/integration/test_migration_0020.py` (task 01, the house migration-suite pattern: upgrade from 0019 adds the columns with the server defaults — a pre-existing row reads `created_at ≈ now()` and `created_at_manual is False`; downgrade restores 0019's schema); `tests/integration/test_git_file_dates.py` (task 03 — real `git` scratch repos through `clone_or_pull` + `file_commit_dates`, both checkout kinds); `tests/integration/test_importer_dates.py` (task 04 — real Postgres: a backfilled-today row refreshed to its old utime on an unchanged re-import; the manual row untouched; the `dates_updated` counter; the date-only refresh does NOT bump `sources_meta`); `tests/integration/test_docs_api_dates.py` (task 05 — `created_at` in both reads; the PATCH matrix: set (round-trip + flag), malformed 422, null-clear (flag drops, date stands), 404 unknown pair, anonymous 403; the tree's `updated_at` on a nested fixture); `tests/integration/test_agent_tools_dates.py` (task 06 — the `ls` line + `read` result carry the date against real rows); `tests/integration/test_recency_boost.py` (task 07 — the fine-line battery, the owner's scenario, real Postgres + axis vectors); `tests/e2e/test_document_dates.py` (task 10, A16, isolation). +- E2E (mandatory, A16) — `uv run pytest tests/e2e/test_document_dates.py -v --no-cov` with the DB up. +- Coverage: **>90%** on `app/` (`uv run pytest --cov=app --cov-report=term-missing` — the validate.sh gate; every new/modified module is fully covered by the suites above). + +## Completion Criteria +- [ ] Every `documents` row has a non-null `created_at` (fresh imports carry the sourced date — git last-commit or mtime; existing deployments' rows read the migration day until the first sync refreshes them); the `0020` migration upgrades from `0019` and downgrades cleanly on the dev/test DB. +- [ ] A sync (button OR CLI) refreshes dates: an unchanged file whose mtime moved gets the new date (`dates_updated` counts it); a file's date may go OLDER; a manually corrected date (`created_at_manual`) survives the sync while its siblings refresh; a content change resets the date and clears the manual flag; a date-only refresh leaves `sources_meta.version`, the overview, and the folder summaries untouched. +- [ ] A zip and a tar with old member timestamps unpack to files carrying those mtimes (the upload path's dates survive the archive process); a file with a future (beyond-skew) or missing date lands as today. +- [ ] The LLM sees the date on every document surface: the HIGH prompt's `` block (mock-LLM regex updated, tolerant), the `read` result's `date:` second line (first line byte-identical), and the `ls` file line's appended ` | date: YYYY-MM-DD` — with the ls↔tree cross-check still holding. +- [ ] The UI shows the date everywhere asked: the file table's `Created` column before `Indexed`; the folder/source table's `Updated` column after `Documents`, before `Description` (subtree max, `–` for empty); the clicked document's top meta row carries the `Created ` badge in BOTH the modal and the full page. +- [ ] An admin edits a document's date in the viewer (set + revert-to-sync), the change round-trips through `PATCH /api/documents/date`, and the next sync preserves the correction while refreshing the rest; non-admins see the byte-identical badge row with no editor; the failure path reverts + announces. +- [ ] Retrieval: the owner's scenario is pinned — an older document that answers the question ranks above a newer similar one that doesn't (defaults AND with the boost off); near-ties break toward the newer document (boost on) and toward the pre-phase order (boost off); `BOR_RECENCY_BOOST=0` restores byte-identical ranking. +- [ ] `uv run pytest` green; `uv run pytest --cov=app --cov-report=term-missing` TOTAL >90%; `uv run pytest tests/e2e/test_document_dates.py -v --no-cov` green in isolation (DB up); the regression suites listed in task 10 green in isolation; `uv run ruff check . && uv run pyright` clean. +- [ ] One `--no-gpg-sign` commit; phase dir moved to `.agents/phases/complete/` by the pipeline gate. + +## Locked decisions +- **D1 — NOT-NULL storage + manual flag (owner-instructed 2026-09-13; flag = the phase-97 `manually_edited` precedent).** `documents.created_at` NOT NULL, server-defaulted to `now()` at migration time (existing deployments get "today", corrected on the next sync — the owner's exact words); `documents.created_at_manual` (default false) protects an owner-corrected date from sync refreshes — owner corrections are never silently rewritten (the phase-97 decision, applied to the document date). +- **D2 — Date provenance (owner-instructed).** Git sources: the per-file LAST-COMMIT date (`git log --name-only --format=@@%cI`, committer date, one git call per source per sync, fail-soft to mtime). Everything else (local dirs, unpacked uploads): the file mtime — with the unpackers fixed to PRESERVE member mtimes (the owner's "hopefully" made real). +- **D3 — Normalization (owner-instructed).** Undetermined → today; future (beyond a 1-day clock-skew tolerance) → today; naive → UTC; all stored as UTC `timestamptz` at full precision (the UI formats). One choke point (`app/rag/doc_dates.py`). +- **D4 — Sync semantics (owner-instructed).** The date refreshes on every sync, including unchanged files, and may go OLDER (no monotonic guard). Added/updated content resets the date + clears the manual flag. A date-only refresh is `unchanged` for every existing gate (no `sources_meta` bump, no overview/summary regeneration, no staleness) and is counted in a NEW additive `dates_updated` counter (log line + sync detail). +- **D5 — LLM surfaces (owner-instructed: "all documents must include the date when fed to the LLM").** The `date="YYYY-MM-DD"` attribute appended to the `` block (after `title`); the `read` result's second line `date: YYYY-MM-DD` (first line byte-identical — the mock header contract); the `ls` FILE line's appended ` | date: YYYY-MM-DD` (appended, never inserted — the mock line regex). No persona/teaching copy changes. +- **D6 — The recency boost (owner-instructed: "ranked higher… or at least given a boost, without breaking the existing retrieval process").** An additive post-fusion term on the RRF score — `fused + recency_boost·exp(−age_days/recency_half_life_days)` — defaults `0.001` / `365` days, env-tunable (`BOR_RECENCY_BOOST`, `BOR_RECENCY_HALF_LIFE_DAYS`), `0` = byte-identical off, negatives fail startup loudly (the house validator pattern). Applied once in `retrieve()` after `fuse()`. The A7 math, the A8 cosine gate, `query_log.top_score`, and the never-truncated contract are untouched. The owner's old-correct-beats-new-similar scenario is a permanent integration pin. +- **D7 — Editing (owner-instructed: "editable so users can correct for errors").** Admin-only `PATCH /api/documents/date` (the phase-57 split: the viewer stays user-gated, the edit is admin-gated); a date string sets + flags manual; null clears the flag (the date stands until the next sync). The editor lives in the shared viewer core (modal + page), phase-57 idiom, §7.4 never-stale lifecycle. +- **D8 — UI columns/badge (owner-instructed positions, verbatim).** Files: `Created` between `Chunks` and `Indexed`. Folders/sources: `Updated` between `Documents` and `Description`. Clicked document: the `Created` badge in the top meta row, before `Indexed`. +- **D9 — Folder dates are derived.** No folder date storage: `updated_at` = the subtree's max document `created_at`, computed in the pure tree builder (one concept end to end — the same recursion that counts `documents`); `None` for an empty source. +- **D10 — No clone-strategy change (phase 28 stands).** `--depth 1` stays. Consequence (verified 2026-09-13): local-path git sources clone with FULL history → true per-file dates; URL git sources are shallow → every file carries the repo's TIP-commit date (uniform within the repo — no intra-repo distortion, a real cross-source signal, refreshed on every pull). Revisit only if the owner later wants intra-repo recency on URL sources. + +**A7 note (PLAN anchor, owner-permitted extension 2026-09-13):** the recency boost is an additive POST-fusion re-ranking on A7's fused score, requested by the owner in this phase's ask ("newer documents should be ranked higher in retrieval… without breaking the existing retrieval process"). A7's hybrid mechanics, windows, k, and never-truncated contract are unchanged; when the owner signs off, `PLAN.md` §2's A7 row gains a dated revision note (the skill that authors phases never edits `PLAN.md`). + +## Commit +```bash +git add app/ alembic/versions/0020_documents_created_at.py scripts/ frontend/ tests/ .env.example .agents/phases/ && git commit --no-gpg-sign -m "feat(dates): document dates end to end — sourced at sync, shown in UI, editable, recency-weighted in retrieval" +``` diff --git a/.agents/phases/todo/106_document_dates/01_created_at_column.md b/.agents/phases/todo/106_document_dates/01_created_at_column.md new file mode 100644 index 0000000..66e78d6 --- /dev/null +++ b/.agents/phases/todo/106_document_dates/01_created_at_column.md @@ -0,0 +1,50 @@ +# Task 01 — `documents.created_at` + `documents.created_at_manual` (model + alembic `0020`) + +**Phase:** `106_document_dates` · **Source:** owner request 2026-09-13 — "Date should be stored in the bor database"; "the timestamp can't be null so just set it to today's date during the migration"; "This timestamp should be editable" (the manual flag is D1's phase-97-precedent half). + +## Objective +Persist the document creation date: one additive, reversible migration adding `created_at` (NOT NULL, server-defaulted to the migration moment — every existing deployment row reads "today") and `created_at_manual` (default false — the owner-correction lock, D1) to `documents`. + +## Work +1. `app/models.py` — the `Document` class (L101-130): add the two columns directly AFTER `indexed_at` (L112), mirroring its docstring/provenance style (`DateTime`/`Boolean`/`func`/`text` are already imported): + ```python + #: The document's CREATION date (phase 106, D1/D2/D3) — sourced at + #: sync time (git last-commit date for git sources, file mtime for + #: local dirs / unpacked uploads), normalized by + #: :func:`app.rag.doc_dates.normalize_doc_date` (undetermined or + #: future → today; UTC). NOT NULL: pre-phase-106 rows backfill to + #: the migration moment (≈ today — the owner's instruction) and the + #: next sync refreshes them (the importer's unchanged path, + #: task 04 — a sync may move a date OLDER, D4). Distinct from + #: ``indexed_at`` (the INDEX time, untouched). + created_at: Mapped[datetime] = mapped_column( + DateTime(timezone=True), server_default=func.now(), nullable=False + ) + #: True only while ``created_at`` is the OWNER'S correction (phase + #: 106, D1 — the ``folder_summaries.manually_edited`` phase-97 + #: precedent): set ONLY by ``PATCH /api/documents/date`` + #: (task 05); the sync-time importer SKIPS the refresh on a manual + #: row (the correction survives syncs, D4) and a content change + #: RESETS both the date and the flag (a new version = a new date). + created_at_manual: Mapped[bool] = mapped_column( + Boolean, default=False, server_default=text("false"), nullable=False + ) + ``` + (If the module header's one-line `documents` field inventory names `indexed_at`, add `created_at`/`created_at_manual` (phase 106) to the parenthetical.) +2. `alembic/versions/0020_documents_created_at.py` (NEW — the house format of `0019_git_source_include_hidden.py`): + - `revision = "0020"`, `down_revision = "0019"`. + - `upgrade()`: `op.add_column("documents", sa.Column("created_at", sa.DateTime(timezone=True), server_default=sa.text("now()"), nullable=False))` then `op.add_column("documents", sa.Column("created_at_manual", sa.Boolean(), server_default=sa.text("false"), nullable=False))`. + - `downgrade()`: `op.drop_column("documents", "created_at_manual")` then `op.drop_column("documents", "created_at")`. + - Module docstring: the phase-106 provenance (what the date is, D1/D2/D3/D4, the NOT-NULL backfill-to-today behavior, one additive reversible migration, A13). +3. Tests — `tests/integration/test_migration_0020.py` (NEW), mirroring `tests/integration/test_migration_0019.py` VERBATIM in shape (the real-Alembic `alembic` fixture that starts/ends at head; `information_schema` column-contract assertions; the explicit 0019 → 0020 step so later migrations cannot break the pins): the 0019 `documents` schema (incl. `indexed_at`, `summary`) survives the upgrade; both new columns exist with the full contract — `timestamp with time zone` NOT NULL default `now()` / `boolean` NOT NULL default `false`; a `documents` row inserted while the DB is at `0019` backfills `created_at ≈ now()` (assert within a few seconds of the upgrade moment) and `created_at_manual is False`; downgrade to `0019` → both columns GONE (A13) while the row + its content survive; upgrade back to `0020` → both columns back (round-trip); the ORM contract agrees — a freshly inserted `Document` (nothing passed) reads `created_at_manual is False` + non-null `created_at`, and an explicit `created_at` + `created_at_manual=True` round-trips through a fresh session. +4. Run `uv run pytest tests/integration/test_migration_0020.py -q` (DB up) + `uv run alembic upgrade head` on the dev/test DB — green. + +## Testing & Quality +- Integration: the migration upgrade/downgrade + server-default pins above ARE this task's layer (no importer behavior yet — task 04 writes these columns). +- Coverage: **>90%** on `app/` (model/migration-only change — the validate.sh gate). + +## Completion Criteria +- [ ] `Document.created_at` (NOT NULL, `server_default=func.now()`) and `Document.created_at_manual` (NOT NULL, `server_default=text("false")`) exist with the D1/D2/D3/D4 provenance comments +- [ ] `alembic/versions/0020_documents_created_at.py` upgrades from `0019` and downgrades cleanly; the dev/test DB is at head; existing rows read `created_at ≈ now()` (the backfill) and `created_at_manual is False` +- [ ] Fresh-row-defaults + explicit-values round-trip tests pass; existing suites stay green +- [ ] `uv run ruff check . && uv run pyright` clean diff --git a/.agents/phases/todo/106_document_dates/02_date_normalization.md b/.agents/phases/todo/106_document_dates/02_date_normalization.md new file mode 100644 index 0000000..2b271d7 --- /dev/null +++ b/.agents/phases/todo/106_document_dates/02_date_normalization.md @@ -0,0 +1,73 @@ +# Task 02 — `app/rag/doc_dates.py`: the date normalization choke point (D3) + +**Phase:** `106_document_dates` · **Source:** owner request 2026-09-13 — "If the date of a document can't be determined or is in the future then assume that document was created today." + +## Objective +One pure, stdlib-only module that every document date passes through — the importer (task 04) and the date-edit API (task 05) both call it, so the today/future/naive rules live in exactly one place and can be pinned by unit tests without a database. + +## Work +1. `app/rag/doc_dates.py` (NEW): + ```python + """Document-creation-date sourcing + normalization (phase 106, D2/D3). + + Every document date the importer writes and every date the owner + edits passes through :func:`normalize_doc_date` — the single choke + point for the owner's rules: an UNDETERMINED date (no source signal) + and a FUTURE date (beyond a small clock-skew tolerance) both assume + the document was created TODAY (UTC). Naive source timestamps (zip + DOS mtimes, tar mtimes, git-free fallbacks) are tz-agnostic epoch- + based values rendered as UTC; aware ones are converted to UTC. + """ + from __future__ import annotations + + from datetime import UTC, datetime, timedelta + from pathlib import Path + + #: Clock-skew tolerance (D3): a source date up to this far in the + #: FUTURE is a drifting clock, not a future document — it keeps its + #: date. Beyond it, the owner's rule applies (→ today). + FUTURE_SKEW_TOLERANCE = timedelta(days=1) + + + def normalize_doc_date(raw: datetime | None, now: datetime | None = None) -> datetime: + """*raw* → the stored UTC creation date (the D3 rule, pinned). + + ``now`` is injectable (tests); it defaults to + ``datetime.now(UTC)``. ``raw=None`` (undetermined) → *now*; + naive *raw* → treated as UTC; aware *raw* → converted to UTC; + *raw* beyond *now* + :data:`FUTURE_SKEW_TOLERANCE` → *now*. + The result always carries full precision (no date-truncation — + the display formats, the storage doesn't). + """ + ``` + Plus: + ```python + def file_mtime_datetime(path: Path) -> datetime: + """The file's mtime as an aware UTC datetime (the D2 fallback). + + Epoch mtimes are tz-agnostic — UTC is the correct rendering + (zip DOS timestamps and tar mtimes pass through the same + :func:`normalize_doc_date` after unpacking, task 03). + """ + return datetime.fromtimestamp(path.stat().st_mtime, tz=UTC) + ``` + Implementation notes: for the naive case, attach UTC (`raw.replace(tzinfo=UTC)`) rather than assuming local time (the homelab host TZ is irrelevant — source mtimes are epoch values); for the aware case, `raw.astimezone(UTC)`; compare the future check in aware space. +2. `tests/unit/test_doc_dates.py` (NEW) — the boundary matrix (pure function, no DB): + - `None` → exactly `now` (inject a fixed `now`); + - naive `2020-05-01T12:00` → `2020-05-01T12:00+00:00` (UTC-attached, not local-converted); + - aware `2020-05-01T08:00-04:00` → `2020-05-01T12:00+00:00` (converted); + - future by 23 h (just INSIDE the tolerance) → keeps its date; + - future by 25 h (beyond) → `now`; + - exactly `now + FUTURE_SKEW_TOLERANCE` → keeps its date (the boundary is strict-greater); + - `file_mtime_datetime` on a tmp file with a `os.utime`'d mtime → the expected UTC datetime (±1 s tolerance for mtime granularity); + - the module imports nothing but stdlib (a source-level pin, the house pattern — grep the file for `import` lines). +3. Run `uv run pytest tests/unit/test_doc_dates.py -q` — green. + +## Testing & Quality +- Unit: the matrix above IS this task's layer (the callers land in tasks 04/05). +- Coverage: **>90%** on `app/` (new module fully covered — the validate.sh gate). + +## Completion Criteria +- [ ] `app/rag/doc_dates.py` exists with `FUTURE_SKEW_TOLERANCE` (1 day), `normalize_doc_date` (None→today, naive→UTC, aware→convert, future-beyond-tolerance→today, full precision kept) and `file_mtime_datetime` +- [ ] `tests/unit/test_doc_dates.py` pins the full boundary matrix (incl. the strict-greater 1-day boundary and the naive-is-UTC rule) and passes +- [ ] No non-stdlib imports in the module; `uv run ruff check . && uv run pyright` clean diff --git a/.agents/phases/todo/106_document_dates/03_source_date_extraction.md b/.agents/phases/todo/106_document_dates/03_source_date_extraction.md new file mode 100644 index 0000000..df70315 --- /dev/null +++ b/.agents/phases/todo/106_document_dates/03_source_date_extraction.md @@ -0,0 +1,38 @@ +# Task 03 — Source date extraction: mtime-preserving unpack + `file_commit_dates` (D2/D10) + +**Phase:** `106_document_dates` · **Source:** owner request 2026-09-13 — dates come "via their git timestamp or via file metadata (hopefully) preserved in the tar or zip archive process." + +## Objective +Make the source-of-truth date actually EXIST at the filesystem/checkout level: the zip/tar unpacker restores member mtimes (uploads stop losing their dates), and `scripts/git_sync.py` gains a single-call per-file last-commit-date walk (the A11 git site) with the verified shallow-vs-local behavior pinned. + +## Work +1. `app/rag/archive_upload.py` — mtime preservation (D2), regular files only, every safety check + the zip-bomb cap UNCHANGED: + - `_unpack_zip` (L214-238): after the `with zf.open(member) as src: _write_capped(...)` block for a regular file, restore the member's DOS mtime: + ```python + mtime = datetime(*member.date_time, tzinfo=timezone.utc).timestamp() + os.utime(dest, ns=(mtime, mtime)) + ``` + (`datetime`/`timezone` from the stdlib — add the import; DOS `date_time` is a tz-agnostic epoch value, UTC-rendered exactly like an mtime — the task-02 convention.) + - `_unpack_tar` (L241-270): in the `member.isreg()` branch, after `_write_capped(...)`: `os.utime(dest, ns=(member.mtime, member.mtime))` (tar `mtime` is epoch seconds — `ns=` accepts a float seconds value). + - Directories, symlinks, and hardlinks are untouched (only regular files are ever indexed). A failed unpack still removes the partial tree (the `utime` calls sit inside the existing try/except flow — an `OSError` there is caught by `unpack_archive`'s handler exactly like any other write failure). + - Update the module docstring's guarantees list with the mtime-preservation line (phase 106, D2). +2. `scripts/git_sync.py` — `file_commit_dates(dest: Path) -> dict[str, datetime]` (NEW public function, exported in `__all__`): + - Runs ONE `run_git(["git", "log", "--name-only", "--format=@@%cI"], cwd=dest)` (the A11 single-invocation site — the module docstring's git-inventory sentence gains this command). + - Parse: lines matching `@@` start a commit (ISO-strict `%cI` → `datetime.fromisoformat`, aware); subsequent non-empty lines until the next `@@`/blank-then-`@@` are repo-relative paths (split on whitespace like git's name-only output, normalize `\` → `/`, lstrip a leading `/`). Per path, the FIRST sighting wins (the walk is newest-first) — that is the file's last-commit date. + - **Fail-soft (pinned):** `GitSyncError` (git missing/failed) or ANY parse anomaly → `logger.warning` + return `{}` — the importer (task 04) falls back to file mtimes; a date walk must never break a sync. + - Module docstring: what it is, the one-git-call contract, and the VERIFIED checkout behavior (owner-permission source: this phase's ask, 2026-09-13): a local-path checkout cloned by `clone_or_pull` keeps FULL history (`--depth` is ignored in local clones — git's own warning) → TRUE per-file dates; a URL-transport checkout is shallow and git reports the TIP commit as every existing file's last commit (the shallow boundary is each file's history root) → a uniform per-repo tip date (D10 — no intra-repo distortion, real cross-repo signal). +3. Tests: + - `tests/unit/test_archive_upload_dates.py` (NEW): build in `tmp_path` — a zip with one member whose `ZipInfo.date_time` is an old fixed tuple (e.g. `(2020, 1, 2, 3, 4, 6)` → 2020-01-02 03:04:06 UTC) and a tar with one member `mtime=1577934246` (2020-01-02 03:04:06) — `unpack_archive` → the extracted file's `st_mtime` equals the member's (±1 s, mtime granularity). The existing archive-upload suite (`tests/unit/test_archive_upload*.py` — glob to find it) stays green (no safety behavior moved). + - `tests/integration/test_git_file_dates.py` (NEW — real `git` in the test environment, the `test_import_docs_git.py` precedent for git availability; skip cleanly if `git` is absent, that suite's pattern): in `tmp_path_factory` build a scratch repo with two files committed at controlled `GIT_COMMITTER_DATE`s (file A 2020-01-02, file B touched again 2024-06-15 — the 2026-09-13 verification recipe): (a) `clone_or_pull` from the LOCAL path → `file_commit_dates` returns A's 2020 date and B's 2024 date (true per-file); (b) a `file://` shallow clone (run `git clone --depth 1 file://…` directly in the test — the test harness, not `clone_or_pull`, makes this one) → EVERY file's date is the TIP commit's (2024-06-15) (D10 pinned); (c) a directory without `.git` / a `git` failure → `{}` (fail-soft, no raise). +4. Run `uv run pytest tests/unit/test_archive_upload_dates.py tests/integration/test_git_file_dates.py -q` (DB up for the integration file's `db` fixture only if used — keep it DB-free: `file_commit_dates` takes a path, no session) — green. + +## Testing & Quality +- Unit: the zip/tar mtime pins + the safety-suite regression. +- Integration: the git walk against real scratch repos (both checkout kinds + the fail-soft path) — DB-free. +- Coverage: **>90%** on `app/` (the unpacker branches + the new parser fully covered — the validate.sh gate; `scripts/` is outside the `app/` coverage denominator but the integration suite pins its behavior). + +## Completion Criteria +- [ ] A zip and a tar with old member timestamps unpack to files carrying those mtimes (regular files only; safety/cap behavior byte-identical — the existing suite green) +- [ ] `scripts/git_sync.py::file_commit_dates` exists, is the ONLY new git invocation (through `run_git`), returns `{path: last_commit_datetime}` with first-sighting-wins parsing, and fails soft to `{}` +- [ ] The verified behavior is pinned: local clone → true per-file dates; `file://` shallow clone → tip date for every file +- [ ] `uv run ruff check . && uv run pyright` clean diff --git a/.agents/phases/todo/106_document_dates/04_importer_dates.md b/.agents/phases/todo/106_document_dates/04_importer_dates.md new file mode 100644 index 0000000..5bf774f --- /dev/null +++ b/.agents/phases/todo/106_document_dates/04_importer_dates.md @@ -0,0 +1,43 @@ +# Task 04 — Importer: source the date on every upsert, refresh on unchanged, protect manual (D4) + +**Phase:** `106_document_dates` · **Source:** owner request 2026-09-13 — "Date should be stored in the bor database and updated when sources are synced"; "It's totally fine for a sync to cause a document or folder's date to get older." + +## Objective +The importer writes `documents.created_at` from the source (git map → mtime fallback → `normalize_doc_date`) on every add/update, REFRESHES it on the unchanged path (the backfill-correction case — an existing row stamped "today" by the migration gets its real date on the next sync even when the content didn't change), skips the refresh on manually corrected rows, and counts date-only refreshes in a new additive `dates_updated` counter. Both live entry points (Sync button, CLI) feed the git-date map. + +## Work +1. `app/rag/importer.py`: + - `ImportSummary` (L64-111): add `dates_updated: int = 0` (docstring: files whose `created_at` was refreshed on the UNCHANGED path — content untouched, D4) and the `dates_updated=%d` term in `log()`'s `import: summary …` line (PLAN §9 — append it AFTER `summary_errors`, before `formats`, so existing prefix assertions survive). + - `import_sources` (L213-340): new keyword `doc_dates_by_root: dict[str, dict[str, datetime]] | None = None` (AFTER `include_hidden_by_root`), docstring paragraph in the `include_hidden_by_root` style: keyed by `str(root)` — the root string exactly as passed in *sources*; maps a source-relative POSIX path to its RAW source date (git last-commit, task 03); ONLY git roots are listed — unlisted roots (local dirs, uploads) take the mtime fallback; `None` (default) changes nothing for existing callers (the mtime fallback applies — which IS the behavior change: unchanged files now refresh their date, D4). In the processing loop (L281+): `dates_map = (doc_dates_by_root or {}).get(str(root), {})` and pass `raw_date=dates_map.get(rel)` into `_index_file`. The progress pre-walk is UNTOUCHED (dates change no file count). + - `_index_file` (L376-471): new keyword `raw_date: datetime | None = None`: + - Resolve once, up top: `if raw_date is None: raw_date = file_mtime_datetime(full_path)` (import from `app.rag.doc_dates`). + - **added branch** (L409-417): `created_at=normalize_doc_date(raw_date)` on the new `Document(…)`; `created_at_manual` stays the column default (`False`). + - **updated branch** (L418-421): `doc.created_at = normalize_doc_date(raw_date)` and `doc.created_at_manual = False` (a content change resets a previous correction — the correction referred to the old content; D4). + - **unchanged branch** (L401-404, currently the early return): BEFORE returning — if `doc.created_at_manual` → return unchanged (log the existing line, the correction survives — D1/D4); else `target = normalize_doc_date(raw_date)`; if `target != doc.created_at` → `doc.created_at = target`, `session.commit()`, `summary.dates_updated += 1`, `logger.info("import: date-refreshed source=%s path=%s date=%s", source, rel, doc.created_at.isoformat())`; return. (A date-only refresh is still counted `unchanged` — `added/updated/pruned` are untouched → no `sources_meta` bump, no overview/folder-summary regeneration: the gate keys on content, D4.) + - Module docstring: the Scope/workflow paragraph gains the date rule (two sentences — sourced on add/update, refreshed on unchanged unless manual, D2/D4). +2. `app/api/sync.py` — `_run_sync` (the per-row loop L233-263): build `doc_dates_by_root: dict[str, dict[str, datetime]] = {}` alongside the other two maps; for `kind=git` rows, AFTER `clone_or_pull` returns: `doc_dates_by_root[str(root)] = file_commit_dates(root)` (import `file_commit_dates` next to the existing `clone_or_pull` import, L100); local rows add nothing (mtime fallback). Pass `doc_dates_by_root=doc_dates_by_root` to `import_sources` (L300-303). The success `detail` dict (L370-382) gains `"dates_updated": summary.dates_updated` (additive key, after `"summary_errors"`). The module docstring's pipeline step 4 gains the third-map clause. +3. `scripts/import_docs.py` — `_resolve_sources` (L168-226): build the same map for the git rows it clones (after the `clone_or_pull` call, L220) and return it as a 4th tuple element `(sources, ignore_by_root, include_hidden_by_root, doc_dates_by_root)` — manual `--source` dirs and env-fallback rows contribute nothing (no row, no clone → no map entry → mtime fallback); update the return docstring. `main` unpacks the 4-tuple (the L269-ish unpack) and passes the map to `import_sources` (L331-334). Module docstring updated. +4. `scripts/load_test_kb.py` — untouched (the `None` default). +5. Tests: + - `tests/unit/test_importer_dates.py` (NEW — the `tests/unit/test_importer_include_hidden.py` scaffolding: fake LLM from `tests/fakes.py` + a tmp fixture tree; run against the `db` session the house unit pattern uses for importer tests — read `test_importer_include_hidden.py` first and mirror its session handling): + - a file `os.utime`'d to 2020-01-02 imports with `created_at` ≈ that instant (added); + - unchanged re-import with the mtime moved to 2021 → `created_at` refreshed, `summary.unchanged == 1` AND `summary.dates_updated == 1` (content counts preserved); + - unchanged re-import with the same mtime → `dates_updated == 0`; + - a row with `created_at_manual=True` + moved mtime → date UNTOUCHED (the D1 lock) and `dates_updated == 0`; + - a content change on a manual row → date reset from source AND `created_at_manual is False`; + - `doc_dates_by_root` map entry beats the mtime (the git case: map says 2020, mtime says now → 2020 stored); + - a future mtime (2030) → `created_at` folds to today (D3 through the importer). + - `tests/integration/test_importer_dates.py` (NEW — real Postgres, the `tests/integration/test_importer_e2e.py` fake-LLM pattern): the backfill-correction case — a row first imported with a "today" mtime, its file then `os.utime`'d back to 2019 (content identical) → the second `import_sources` run stores the 2019 date (`added/updated/pruned` all 0, `dates_updated == 1`) AND `sources_meta`'s version is UNBUMPED (the date-only-refresh gate, D4 — seed the version row first, read it after); a pruned/manual matrix as needed for coverage. + - Regression sweep (run, and update ONLY exact-string pins that break — the `import: summary` line gained a term and the sync `detail` gained a key): `uv run pytest tests/unit/test_importer*.py tests/integration/test_importer*.py tests/integration/test_sync_api.py tests/integration/test_import_docs_git.py -q`. +6. Run the full unit + integration importer slice — green. + +## Testing & Quality +- Unit: the semantic matrix above (added/updated/unchanged × manual × map vs mtime × future) against the fake LLM. +- Integration: real Postgres for the backfill-correction + no-version-bump pins. +- Coverage: **>90%** on `app/` (the new branches in `importer.py` + the sync detail all covered — the validate.sh gate). + +## Completion Criteria +- [ ] `import_sources` accepts `doc_dates_by_root` (str(root)-keyed, git-only, None = byte-identical for existing callers); `_index_file` sources added/updated dates from the map → mtime fallback → `normalize_doc_date` and resets `created_at_manual` on content change +- [ ] The unchanged path refreshes `created_at` (date may go OLDER — no monotonic guard), counts it in `dates_updated` (new field + log term), skips manual rows, and NEVER counts toward `added/updated/pruned` (no `sources_meta` bump, no overview/summary regeneration) +- [ ] The Sync button and the CLI both feed the map (git rows only, after clone); `scripts/load_test_kb.py` untouched; the success sync `detail` carries `dates_updated` +- [ ] The regression slice above is green (exact-string log/detail pins updated in place where they break); `uv run ruff check . && uv run pyright` clean diff --git a/.agents/phases/todo/106_document_dates/05_date_apis.md b/.agents/phases/todo/106_document_dates/05_date_apis.md new file mode 100644 index 0000000..26a5098 --- /dev/null +++ b/.agents/phases/todo/106_document_dates/05_date_apis.md @@ -0,0 +1,45 @@ +# Task 05 — Date API surface: reads, the admin date edit, and the tree's dates (D7/D8/D9) + +**Phase:** `106_document_dates` · **Source:** owner request 2026-09-13 — "This timestamp should be editable so users can correct for errors"; the catalog needs the file dates + folder last-updated (the UI in task 08 renders exactly what this task serves). + +## Objective +Serve the date everywhere the UI (task 08) and the viewer need it — `GET /api/docs`, `GET /api/documents/content`, and `GET /api/docs/tree` (files: `created_at`; folders/sources: the derived subtree-max `updated_at`, D9) — and add the admin-only `PATCH /api/documents/date` (set + revert, the phase-57 gate/idiom, D7). + +## Work +1. `app/schemas.py`: + - `DocSummary` (L228-235): add `created_at: str` (ISO-8601 — the `indexed_at` docstring style: "verbatim from the row"). + - `DocContent` (L343-358): add `created_at: str` (after `summary`, before `content` — group the metadata). + - NEW `DateUpdate` (the `SummaryUpdate` shape, L361-373): `source: str`, `path: str`, `date: str | None` (docstring: an ISO date `YYYY-MM-DD` or full ISO datetime; **null/absent = the CLEAR** — drop the manual flag, the stored date stands until the next sync refresh; a malformed non-null value 422s through Pydantic… correction: `str` passes any string — the handler parses (step 3); the 422 comes from the handler, not the model, so the error detail can name the field). + - NEW `DateResult`: `source: str`, `path: str`, `created_at: str`, `created_at_manual: bool` (echoes the stored state — the viewer re-renders from it). + - `KbTreeFile` (L241-256): add `created_at: str` (verbatim from the catalogue row — the `indexed_at` field's docstring pattern). + - `KbTreeFolder` (L259-290) and `KbTreeSource` (L293-320): add `updated_at: str | None` (docstring: the subtree's MAX document `created_at` — D9, derived, never stored; `null` for a 0-document source, the `summary: str | None` shape). +2. `app/api/docs.py`: + - `list_indexed_documents` (L80-119): add `Document.created_at` to the select AND the `group_by` (the `indexed_at` twin, L104/L107); `DocSummary(..., created_at=row.created_at.isoformat())`. + - `get_document_content` (L121-162): `created_at=doc.created_at.isoformat()` in the `DocContent` (L161 site). + - NEW `PATCH /api/documents/date` (route order: next to `update_document_summary`, L164-232 — `require_admin` dependency, the phase-57 gate): + ```python + @router.patch("/documents/date", response_model=DateResult) + def update_document_date(payload: DateUpdate, db: Session = Depends(get_db), + _admin: None = Depends(require_admin)) -> DateResult: + ``` + Logic (DB-only — the `/documents/content` row-lookup rule, no filesystem, no LLM/embedding call — a date is never embedded, the phase-57 no-LLM contrast): look up the row by `(source, path)` → none → 404 `{"detail": "document not found"}` (row-lookup semantics, the traversal-string-is-not-a-row note). `payload.date` truthy → `parsed = datetime.fromisoformat(payload.date)` (a bare `YYYY-MM-DD` and full ISO datetimes both parse; `ValueError` → 422 `{"detail": "date must be an ISO date or datetime (e.g. 2024-06-15)"}`) → `doc.created_at = normalize_doc_date(parsed)` (import from `app.rag.doc_dates` — D3: a manually set FUTURE date also folds to today, consistency with the sourced path) → `doc.created_at_manual = True`. `payload.date` falsy (null/absent — the CLEAR) → `doc.created_at_manual = False` only (the stored date stands; the next sync refreshes it — the API cannot re-read the source, D7). `db.commit()`; return `DateResult` with the stored `created_at.isoformat()` + flag. + - The tree (task-05 half of D8/D9): `TreeDocRow` (L296-302) becomes the 6-tuple `(source, path, title, chunks, indexed_at, created_at)` (both ISO strings — the builder stays pure over plain types); `_folder_counts` / `_level_children` / `_source_node` thread a 6th element through their tuple unpacks (the `_`-named slots gain the date) and `_level_children`/`_source_node` compute each folder/source's `updated_at`: the MAX of the direct files' `created_at` and the children's `updated_at` values (ISO-8601 strings compare correctly lexicographically — they're all the same `isoformat()` shape; document that in the builder docstring) — `None` when the node has no documents at all (the 0-document registered source). `KbTreeFolder(…, updated_at=…)` / `KbTreeFile(…, created_at=…)` / `KbTreeSource(…, updated_at=…)` at their construction sites (L342-375, L467-490). `list_kb_tree` (L492-551): add `Document.created_at` to the query's select + group_by (the `indexed_at` twin, L543-546) and the `doc_rows` comprehension. The `build_kb_tree` docstring gains the D9 clause (updated_at = subtree max, derived, None for empty). +3. Tests: + - `tests/unit/test_kb_tree_builder.py` (extended — the pure builder): file nodes carry `created_at` verbatim; a nested fixture asserts each folder's + the source's `updated_at` = the subtree max (a deeper file's date wins over a shallow sibling's); a 0-document registered source → `updated_at is None` and no children; the ls↔`group_folder_listing` cross-check tests (L210-280) still pass with the extended tuples (task 06 changes the agent side — until then the rows stay 6-tuples on BOTH sides only after task 06; for THIS task the cross-check compares file `(path, title[, chunks, indexed_at])` projections — read the current assertions and keep them green: the tree builder's file tuples are internal to the builder, the cross-check uses the builder's OUTPUT nodes, so it should pass unchanged — verify and pin). + - `tests/integration/test_docs_api_dates.py` (NEW — the `tests/integration/test_docs_api.py` scaffolding: real app + `db` fixture, an admin cookie where that suite gets one): seed two documents in a nested folder (distinct `created_at`s via direct row writes): + - `GET /api/docs` (admin) reports `created_at` per row (and `indexed_at` unchanged); + - `GET /api/documents/content` carries `created_at`; + - `GET /api/docs/tree` — the file node's `created_at` verbatim, the parent folder's and the source's `updated_at` = the max, a registered-but-empty source → `updated_at: null`; + - the PATCH matrix — set `2020-01-02` → 200 + response echoes the stored ISO + `created_at_manual: true` + a re-GET confirms; set a full ISO datetime → accepted; malformed `"not-a-date"` → 422 (the detail names the field); `date: null` → 200 + `created_at_manual: false` + the stored date UNCHANGED; a future date `"2999-01-01"` → stored `created_at` folds to today (D3); unknown `(source, path)` → 404 `document not found`; anonymous → 403 (the gate). +4. Run `uv run pytest tests/unit/test_kb_tree_builder.py tests/integration/test_docs_api_dates.py -q` (DB up) — green. + +## Testing & Quality +- Unit: the pure builder's date threading (max computation, None-for-empty, verbatim file dates). +- Integration: the full API matrix (reads + PATCH set/malformed/clear/future/404/403) against real Postgres. +- Coverage: **>90%** on `app/` (the new route + the builder branches covered — the validate.sh gate). + +## Completion Criteria +- [ ] `GET /api/docs`, `GET /api/documents/content`, and `GET /api/docs/tree` serve `created_at` (files) and `updated_at` (folders/sources — subtree max, `null` when empty, derived in the pure builder, D9) +- [ ] `PATCH /api/documents/date` is admin-only, DB-only, no-LLM: set (ISO date or datetime, future folds to today, `created_at_manual=true`), clear (null → flag drops, date stands), 422 malformed, 404 unknown pair, 403 anonymous — the phase-57 split intact (viewer stays user-gated) +- [ ] `tests/unit/test_kb_tree_builder.py` + `tests/integration/test_docs_api_dates.py` pass; existing docs-API suites stay green +- [ ] `uv run ruff check . && uv run pyright` clean diff --git a/.agents/phases/todo/106_document_dates/06_llm_date_surfaces.md b/.agents/phases/todo/106_document_dates/06_llm_date_surfaces.md new file mode 100644 index 0000000..123fcd0 --- /dev/null +++ b/.agents/phases/todo/106_document_dates/06_llm_date_surfaces.md @@ -0,0 +1,56 @@ +# Task 06 — LLM surfaces: the date rides every document the model sees (D5) + +**Phase:** `106_document_dates` · **Source:** owner request 2026-09-13 — "After this phase, all documents must include the date when fed to the LLM." + +## Objective +The date appears on all three document surfaces the model reads — the HIGH prompt's `` block (the retrieved top-N), the `read` tool result (agent-fetched), and the `ls` file lines (catalog drill) — with the retriever's raw-SQL detached rows gaining the column, the E2E mock's prompt regex updated in lockstep (house rule), and the existing format pins updated mechanically. No persona/teaching copy changes (phase 03 convention). + +## Work +1. `app/rag/retriever.py` — the column plumbing (the vector path returns ORM rows — the column comes for free; only the two raw-SQL detached-row paths need it): + - `_LEXICAL_SQL` (L140-157): add `d.created_at AS created_at,` (after `d.indexed_at`). + - `_NAME_HIT_SQL` (L277-300): same column addition. + - The two detached `Document(…)` reconstructions (L360-372 in `_name_hit_chunks`, L414-424 in `_lexical_candidates`): pass `created_at=row.created_at`. +2. `app/rag/prompts.py` — `build_high_prompt` (L371-376): the block becomes + ```python + blocks = [ + f'\n' + f"{doc.content}\n" + "" + for doc in documents + ] + ``` + (the UTC date part; the attribute APPENDED after `title` — the only position, always present since `created_at` is NOT NULL). Update the function's docstring line describing the block's identity attributes. The deflection path (`build_deflect_prompt` — titles only) is untouched, and its byte-identity pins hold (no documents involved). +3. `app/rag/agent.py` — two surfaces: + - **`read` result** (L1173-1185): the FIRST line stays `Document {doc.source}/{doc.path}:` BYTE-IDENTICAL (the E2E mock's `_READ_RESULT_PREFIX` header contract — `_read_results` strips exactly that header to recover the path); the date is the SECOND line, both the truncated (L1176-1181) and plain (L1183) results: + ```python + f"Document {doc.source}/{doc.path}:\ndate: {doc.created_at:%Y-%m-%d}\n{doc.content[:cap]}\n{TRUNCATION_MARKER}\n…" + ``` + / `f"Document {doc.source}/{doc.path}:\ndate: {doc.created_at:%Y-%m-%d}\n{doc.content}"`. + - **`ls` file line** (appended — NEVER inserted before `title`, where the mock's non-greedy `path` capture would swallow it): `_source_document_rows` (L658-669) returns `(path, title, created_iso_date)` triples (add `Document.created_at` to the select, format `%Y-%m-%d` in the comprehension); `group_folder_listing` (L707-780) — `rows: Sequence[tuple[str, str, str]]`, the file output becomes `(source, path, title, date)` 4-tuples (the subfolder tuples + count are untouched); `render_folder_listing` (L856-905) renders `f"source: {source} | path: {path} | title: {title} | date: {date}"`; the `ls_folder` (L782-796) + `NOT_A_FOLDER` branch (L1121-1130) + `ls_top` source lines are UNCHANGED in shape (source/folder lines carry no date — only FILE lines are documents). Docstrings updated (the `LS_MAX_FILE_LINES` comment's line-format phrase, the module header's L73 format line). +4. `app/api/docs.py` — the ls↔tree cross-check (D9/phase-97 invariant): the docstrings at L326/L376/L461 name the compared shapes — update them to the extended file tuples; `build_kb_tree`'s OUTPUT nodes already carry `created_at` (task 05), so the cross-check test's node-side comparisons gain the date field (step 6). +5. `tests/e2e/mock_llm.py` — `_DOCUMENT_BLOCK_RE` (L785-789): make the date attribute an OPTIONAL group so the mock tolerates pre- and post-phase shapes: + ```python + _DOCUMENT_BLOCK_RE = re.compile( + r'\n(?P.*?)\n', + re.S, + ) + ``` + `title=` docstring comment (L780-784) updated. `_CATALOG_LINE_RE` (L873-875) and `_READ_RESULT_PREFIX` (L867) are UNCHANGED by design (verified: the appended ` | date: …` lands in the greedy `title: .+$` tail; the read first line is byte-identical). +6. Tests + pin updates: + - `tests/unit/test_prompts_dates.py` (NEW): the HIGH block renders `` with the date = the row's UTC date part (inject a fixed `created_at`); the deflection prompt is byte-identical to the pre-phase text for the same inputs (the A8 byte-identity contract holds); the `read` result — both shapes — has the identical first line and the `date:` second line (truncated variant: marker + notice still follow); the `ls` line ends with ` | date: YYYY-MM-DD` and the 50-line cap note is unchanged. + - `tests/integration/test_agent_tools_dates.py` (NEW — the `tests/integration/test_agent_tools.py` scaffolding): real rows with distinct `created_at`s — execute a `read` tool call → result second line = the stored date, first line unchanged; an `ls` drill → every file line carries its date in the appended field. + - **Existing-pin sweep (mechanical, test files only)** — run and update exact-string pins that break: `uv run pytest tests/unit/test_retriever.py tests/unit/test_agent.py tests/unit/test_kb_tree_builder.py tests/integration/test_agent_tools.py tests/integration/test_name_hit_lexical.py tests/integration/test_chat_api.py -q` (the detached-`Document` constructors in test fixtures that set fields explicitly may need `created_at` where the SQL now returns it — the model default covers ORM inserts; raw-SQL projections are app-side, so fixture rows created via the ORM already have the column). +7. Run the sweep + new suites — green. + +## Testing & Quality +- Unit: prompt block / read line / ls line format pins (the byte-identity contracts). +- Integration: the tool surfaces against real rows. +- Coverage: **>90%** on `app/` (the new SQL columns + render branches covered — the validate.sh gate). + +## Completion Criteria +- [ ] The HIGH prompt's `` block carries `date="YYYY-MM-DD"` (after `title`, always present); the deflection prompt stays byte-identical (A8) +- [ ] The `read` result carries `date: YYYY-MM-DD` as its second line (first line byte-identical — the mock header contract); the `ls` FILE line ends with ` | date: YYYY-MM-DD` (source/folder lines unchanged); the ls↔tree cross-check still holds +- [ ] `_DOCUMENT_BLOCK_RE` is date-tolerant (optional group); `_CATALOG_LINE_RE`/`_READ_RESULT_PREFIX` untouched; the mock serves post-phase prompts correctly (a quick smoke: `uv run pytest tests/e2e/test_whole_document_context.py -v --no-cov` green in isolation, DB up) +- [ ] The existing-pin sweep is green (test-file-only updates); `uv run ruff check . && uv run pyright` clean diff --git a/.agents/phases/todo/106_document_dates/07_recency_boost.md b/.agents/phases/todo/106_document_dates/07_recency_boost.md new file mode 100644 index 0000000..fb308f1 --- /dev/null +++ b/.agents/phases/todo/106_document_dates/07_recency_boost.md @@ -0,0 +1,61 @@ +# Task 07 — Recency boost: newer documents rank higher, without breaking retrieval (D6) + +**Phase:** `106_document_dates` · **Source:** owner request 2026-09-13 — "Newer documents should be ranked higher in retrieval somehow, or at least given a boost, without breaking the existing retrieval process (so make sure to test with documents that have the correct answer but are older against documents that are similar and newer but don't quit correctly answer the question). This will be a fine line to walk, so testing is crucial here." + +## Objective +A small, env-tunable, kill-switchable additive recency term on the RRF-fused score — applied once in `retrieve()` after `fuse()` — so a fresh document gets a bounded head start on near-ties while an older document that ACTUALLY answers the question keeps its rank. The owner's scenario is pinned by a permanent real-Postgres battery with deterministic axis vectors. The A7 math, the A8 cosine gate, `query_log.top_score`, and the never-truncated contract are untouched. + +## Work +1. `app/config.py` — the hybrid block (L163-174, after `rrf_k`): + ```python + #: Recency boost on the RRF-fused retrieval score (phase 106, D6): the + #: MAXIMUM additive score a zero-age document gets — + #: ``fused + recency_boost * exp(-age_days / recency_half_life_days)`` + #: (``app.rag.retriever.apply_recency_boost``, applied in + #: ``retrieve()`` after ``fuse()``). ``0`` = off — the pre-phase + #: ranking is byte-identical (the kill switch); negative values fail + #: startup loudly (the ``agent_max_rounds`` validator pattern). + #: 0.001 ≈ a 1-2 rank head start on a 60+ RRF scale — enough to break + #: near-ties toward the newer document, far below the gap between a + #: document that answers and one that merely resembles (the + #: phase-106 fine-line battery pins it). + recency_boost: float = 0.001 + #: Age (days) at which the recency boost halves (phase 106, D6). + #: ``<= 0`` fails startup loudly (same validator family). + recency_half_life_days: int = 365 + ``` + Add the startup validator (find `agent_max_rounds`'s field-validator and follow it — fail loudly naming the field): `recency_boost < 0` → error; `recency_half_life_days <= 0` → error. `.env.example` — document `BOR_RECENCY_BOOST` + `BOR_RECENCY_HALF_LIFE_DAYS` (the hybrid section, the existing comment style). +2. `app/rag/retriever.py`: + - NEW pure function (module-level, next to `fuse`): + ```python + def apply_recency_boost( + chunks: Sequence[RetrievedChunk], + *, + now: datetime | None = None, + weight: float | None = None, + half_life_days: int | None = None, + ) -> list[RetrievedChunk]: + ``` + Defaults from `get_settings()` when omitted; `now` defaults to `datetime.now(UTC)`. For each chunk: `age_days = max(0.0, (now − doc.created_at).total_seconds() / 86400.0)` (a future `created_at` clamps to 0 — consistent with D3's today-folding), `score = score + weight * math.exp(−age_days / half_life_days)` (import `math`; `replace(rc, score=new_score)` — never mutate inputs, the `fuse` convention). Return the list re-sorted with the EXISTING deterministic key `(−score, −cosine, document.path, position)` — with `weight=0` every score is untouched and the order is byte-identical (pinned). Docstring: the D6 contract, the magnitude rationale (0.001 ≈ 1-2 ranks on the k=60 scale — rank 1 vs 2 in one list differs by ~0.00026, rank 1 vs 10 by ~0.0021), the untouched surfaces (A8 gate = cosine, `query_log.top_score` = cosine, `weak_hit_titles` = titles only, the never-truncated top-N), and the single-apply-site rule (`retrieve()` only — chat API + `eval_retrieval` inherit it). + - `retrieve()` (L398-425): after `return fuse(vector, lexical, settings.rrf_k)` → apply: `fused = fuse(...)`; `if settings.recency_boost > 0: return apply_recency_boost(fused)`; `return fused` (weight-0 callers pay nothing). +3. `scripts/eval_retrieval.py` — the printed top-N table gains two columns: the document's `created_at` (UTC date) and the post-boost effective score (the script calls `retrieve()`, which now applies the boost — print both the raw fused and effective where they differ, or just effective + date; keep the verdict column). Docstring line updated. +4. Tests: + - `tests/unit/test_retriever_recency.py` (NEW — fake rows, no DB): age 0 → `+weight` exact; age = half-life → `+weight*exp(-1)` (±1e-9); age 10× half-life → ~`+weight*exp(-10)` (assert `< weight * 1e-3`); future date → full weight (the clamp); `weight=0` → the returned list's `(score, order)` is byte-identical to the input (the kill-switch pin); a tie on raw score breaks toward the newer document; the sort key's `(path, position)` tie-break still applies when scores AND cosines are equal (two docs, same age). + - `tests/integration/test_recency_boost.py` (NEW — real Postgres, `tests/integration/test_name_hit_lexical.py`'s axis-vector idiom VERBATIM: `D=768` unit vectors, exact cosines, `TRUNCATE chunks, documents` fixture, `retrieve()` + `select_documents()` with settings overrides via the house settings-override pattern — check how that suite's siblings inject settings, e.g. `monkeypatch` on `get_settings` or `Settings(_env_file=None, …)`): + 1. **THE OWNER SCENARIO (old-correct beats new-similar).** Question `"How did I configure the backup retention policy?"`. Doc A `backups/retention.md`, `created_at=2020-01-01`: the exact answer — chunk vector = the question vector's axis (cosine 1.0) + its exact tokens in the chunk text (top FTS rank). Doc B `backups/retention-draft.md`, `created_at=yesterday` (the test computes `now − 1d`): topically similar (shares `backup retention policy` tokens — a solid FTS hit at rank 2-3) but a weaker vector (half-parallel axis → cosine ~0.707) and its text says the policy is "under review, no decision yet" (no answer). Assert with DEFAULTS: `select_documents(...)[0].path == "backups/retention.md"` AND the fused (pre-boost, computed via `fuse` directly in the test for the margin) gap A−B ≥ 3× the zero-age boost (record the measured margin in the test docstring — the "comfortable margin" requirement). Assert AGAIN with `recency_boost=0` (settings override): A still first (no-regression pin — relevance alone ordered them). + 2. **The boost is real (near-tie flips toward newer).** Docs C (2019) and D (yesterday) with IDENTICAL chunk text + IDENTICAL vectors (a true tie — same fused score, cosine, FTS rank; the deterministic sort key would otherwise order by path, and path is set so the OLDER sorts first lexicographically, e.g. `c-older.md` < `d-newer.md`). With defaults: D (newer) is first. With `weight=0`: C (older) is first (proving the boost — not drift — is the differentiator). + 3. **Decay end-to-end:** the same C/D pair with D aged to `half_life + 365` days (≈ `weight*e^{-3}` ≈ 0.00005, below the tie gap 0) → C first again (the boost faded — recency is an age signal, not a binary). + 4. **The gate is untouched:** the owner-scenario question's `max cosine` (the A8 input) equals the pre-boost run's (assert on the retrieved chunks' `cosine` values — the boost never touches them). + - If test 1's measured margin under the DEFAULTS is thin (< 3× the boost) or the scenario flips, tune the DEFAULTS (0.001/365 are the starting point — the owner re-tunes live via the env) until old-correct wins comfortably, and record the final margin in the docstring. The test asserts the SEMANTICS (A first, margin ≥ 3× boost), never the exact floats. +5. Run `uv run pytest tests/unit/test_retriever_recency.py tests/integration/test_recency_boost.py -q` (DB up) — green; then `uv run pytest tests/integration/test_name_hit_lexical.py tests/integration/test_chat_api.py -q` (the retriever's existing contract suites stay green — the boost is ON by default in them, so any drift surfaces here). + +## Testing & Quality +- Unit: the decay/weight/clamp/tie/kill-switch pins (pure function). +- Integration: the fine-line battery on real Postgres with exact axis cosines — the owner's scenario + the near-tie flip + the decay + the cosine-gate-untouched pin. +- Coverage: **>90%** on `app/` (config validator + retriever branches covered — the validate.sh gate). + +## Completion Criteria +- [ ] `Settings.recency_boost` (default 0.001, 0 = byte-identical off, negative fails startup) + `recency_half_life_days` (default 365, `<= 0` fails startup); `.env.example` documents both +- [ ] `apply_recency_boost` is pure (defaults from settings, `now` injectable, inputs unmutated, the existing 4-key sort) and is applied in `retrieve()` after `fuse()` and ONLY there — chat API + `eval_retrieval` inherit it; `eval_retrieval` prints the date + effective score +- [ ] The owner's scenario is pinned: older-correct beats newer-similar under defaults (margin ≥ 3× the zero-age boost, recorded) AND with the boost off; the near-tie flips toward the newer with the boost on and back without; the decay pin holds; the A8 cosine input is untouched +- [ ] `tests/unit/test_retriever_recency.py` + `tests/integration/test_recency_boost.py` + the two existing retriever-contract suites green; `uv run ruff check . && uv run pyright` clean diff --git a/.agents/phases/todo/106_document_dates/08_ui_dates.md b/.agents/phases/todo/106_document_dates/08_ui_dates.md new file mode 100644 index 0000000..32b3215 --- /dev/null +++ b/.agents/phases/todo/106_document_dates/08_ui_dates.md @@ -0,0 +1,38 @@ +# Task 08 — UI: `Created` file column, `Updated` folder column, viewer `Created` badge (D8) + +**Phase:** `106_document_dates` · **Source:** owner request 2026-09-13 — "The UI must also show a date for every document at the top of that document when the user clicks it"; "I would also like to see a last updated dates/timestamps on folders before the description column but after the documents column in the UI"; "For files, include a date/timestamp before the 'indexed' column in the UI." + +## Objective +Render the dates the task-05 APIs serve: the file table's `Created` column (before `Indexed`), the folder/source table's `Updated` column (after `Documents`, before `Description`), and the clicked document's `Created` badge in the shared viewer core's top meta row (modal + full page). The date EDITOR is task 09 — this task ships display only. + +## Work +1. `frontend/index.html` — the two header rows (RAG view): + - File table (L485-492): insert `Created` between `Chunks` and `Indexed`. + - Folder table (L470-478): insert `Updated` between `Documents` and `Description`. + (No other shell markup — the rows are built by JS; a brief phase-106 comment above each inserted `` in the house style.) +2. `frontend/assets/sources.js`: + - `makeRow` (L1375-1410): the cell loop (L1401) becomes `for (const value of [d.title, String(d.chunks), fmtDate(d.created_at), fmtDate(d.indexed_at)])` — the `Created` cell lands BEFORE `Indexed` (D8 verbatim). The loop's plain-`td` shape can't carry per-cell titles, so the date cells get one refinement: build the `Created` cell explicitly (a `td` with `textContent = fmtDate(d.created_at)` AND `title = d.created_at` — the ISO hover/precision value, the path-cell `title` idiom) between the `chunks` and `Indexed` cells (the E2E asserts on the locale-stable `title`, not on `toLocaleString` output). The row object fed from tree file nodes (L1341-1350) gains `created_at: f.created_at` (task 05's tree shape — the flat `GET /api/docs` path, if `makeRow` is still fed from it anywhere, carries `created_at` too — grep `makeRow(` call sites and extend every one). + - `makeSourceRow` (L1217-1235) + `makeFolderRow` (L1237-1261): between the count `td` and the description cell, one new `td` — `const updatedTd = document.createElement("td"); updatedTd.textContent = s.updated_at ? fmtDate(s.updated_at) : "–";` (the `statLast` null idiom, L1294 — `None` for a 0-document source, D9). `title` attribute = the ISO value (hover precision on the ellipsized cell, the `makeRow` path-cell idiom). + - `renderLevel`/`treeStats` — UNCHANGED (the stat cards keep their `indexed_at` "last indexed" semantics — the owner asked for the column, not the cards). +3. `frontend/assets/document.js` — `renderDocument` (L118-176, the ONE shared core — the modal AND `/document.html` render through it): the `.doc-meta` badge row (L123-129) gains the badge BEFORE the `Indexed` one: + ```js + metaBadge("doc-created", `Created ${fmtDate(doc.created_at)}`), + metaBadge("doc-indexed", `Indexed ${fmtDate(doc.indexed_at)}`), + ``` + (the date at the top of a clicked document, D8). `doc-created` is the NEW class — the badge's `title` attribute carries the full ISO timestamp (the `titleEl` ellipsis-precision idiom, L122-124). `document-modal.js` needs no change (it calls the shared core with its own `metaEl` — the module docstring's contract is unchanged; verify the modal's meta element exists — it does: `metaEl` L48). +4. `frontend/assets/styles.css` — next to the existing `.doc-indexed` rule (grep for it): `.doc-created` — same badge family (the `doc-indexed` rule copied, provenance comment citing phase 106 D8); the new table cells need no new CSS beyond what `.docs-table` already styles (verify the column count change doesn't break the table's responsive rules — the `#docs-table`/`.kb-folders-table` grid/width rules: if a rule hard-codes the column count, extend it). WCAG: the date text reuses the table ink (≥4.5:1 by construction — record the verified pair in the comment, house style); the badge contrast mirrors `doc-indexed`'s recorded ratio. +5. `tests/unit/test_sources_dates.py` (NEW — the house read-the-assets-as-text pattern, `tests/unit/test_source_ignore_paths.py`'s sibling style): + - `frontend/index.html` — both header rows' cell ORDER pinned (the `` sequence strings: `Source | Path | Title | Chunks | Created | Indexed` and `Folder | Documents | Updated | Description`); + - `frontend/assets/sources.js` — the `makeRow` value-list order (`created_at` before `indexed_at`), the `updatedTd` null→`"–"` branch present in BOTH row builders, the file-row object carries `created_at`; + - `frontend/assets/document.js` — the badge order in the meta row (`doc-created` before `doc-indexed`), the `Created ` label + `fmtDate(doc.created_at)` template, the single-source cross-file check that the `doc-created` class exists in `styles.css`; + - `frontend/assets/styles.css` — the `.doc-created` rule present with a provenance comment. +6. Run `uv run pytest tests/unit/test_sources_dates.py -q` + the existing sources/JS unit suites — green. + +## Testing & Quality +- Unit: the source-level wiring pins above (order, null handling, cross-file class check). +- Coverage: **>90%** on `app/` (no `app/` code this task — the gate is the full-suite one, held by the other tasks; the JS pins are the house frontend-test pattern). + +## Completion Criteria +- [ ] The file table shows `Created` between `Chunks` and `Indexed` (formatted like the `Indexed` cell — `fmtDate`); the folder/source table shows `Updated` between `Documents` and `Description` (subtree max from the tree API, `–` when null) +- [ ] The clicked document's top meta row carries `Created ` BEFORE `Indexed` in BOTH the modal and `/document.html` (one shared core — no per-surface copy) +- [ ] `tests/unit/test_sources_dates.py` pins the orders + null branch + cross-file class and passes; existing JS unit suites stay green; `uv run ruff check . && uv run pyright` clean diff --git a/.agents/phases/todo/106_document_dates/09_date_editor.md b/.agents/phases/todo/106_document_dates/09_date_editor.md new file mode 100644 index 0000000..d9a1cc8 --- /dev/null +++ b/.agents/phases/todo/106_document_dates/09_date_editor.md @@ -0,0 +1,31 @@ +# Task 09 — The admin date editor in the viewer (D7, the phase-57 idiom) + +**Phase:** `106_document_dates` · **Source:** owner request 2026-09-13 — "This timestamp should be editable so users can correct for errors." + +## Objective +An admin-only inline date editor in the shared viewer core (modal + page, where task 08 put the badge): set a corrected date (→ `PATCH /api/documents/date`, `created_at_manual` locks it against syncs, D1) or revert to sync-managed (the CLEAR — flag drops, the date stands until the next sync refreshes). Non-admins see the byte-identical task-08 badge row — no button, no wiring, no network call (the phase-57 split, owner-locked). + +## Work +1. `frontend/assets/document.js` (extend the task-08 core — the phase-57 `wireSummaryEdit` idiom verbatim in structure; read it first, L180+): + - After the `Created` badge (task 08's insertion point), the admin gate: `void docAdminReady().then((admin) => { if (admin) wireDateEdit(metaEl, doc); })` — the module-cached `docAdminReady()` promise (the phase-57/79 single-request-per-page convention — no extra fetch). Anonymous / token holders / a failed whoami: the badge row stays exactly what task 08 built (byte-for-byte). + - `wireDateEdit(metaEl, doc)`: + - **The button** — a text button `Edit date` (`.doc-date-edit`, the `.kb-summary-edit`/`.doc-summary-edit` button family — reuse the existing edit-button class if its styling fits, else a sibling class in `styles.css` with the provenance comment), inserted after the Created badge, `aria-label` = `Edit creation date: ${doc.source}/${doc.path}` (setAttribute — never innerHTML). + - **The editor** (opened on click — the badge row swaps in-place, the summary editor's swap pattern): the `Edit date` button is replaced by a container holding a native `` (value = `doc.created_at`'s UTC date part — `new Date(doc.created_at).toISOString().slice(0, 10)`; `aria-label="Document creation date"`) + `Save` / `Cancel` text buttons + a `role="status"` live line (the phase-57 status-line shape). `Save` with an empty input → the clear path (see below) is NOT implicit — an empty `type=date` input is disabled-look only: disable Save when empty (an explicit `Revert` link below handles the clear — no accidental wipes). + - **Revert affordance** (the D7 CLEAR, the phase-57 "clear = explicit" contrast): a `Revert to sync` text link/button in the editor container (the muted marker style) → sends `{source, path, date: null}`. + - **§7.4 never-stale lifecycle** (the phase-57/89 last-announce order): on Save/Revert — the editor controls disable IMMEDIATELY (no double-submit); `PATCH /api/documents/date` with `{source: doc.source, path: doc.path, date: }` (or `date: null` for the revert); on 200 → the badge's text re-renders from the RESPONSE's `created_at` (`Created ${fmtDate(res.created_at)}` — the UI shows exactly what the server stored, never the input's optimistic value), the status line announces `Date saved for /.` / `Reverted to sync-managed date.` (the `role=status` live line + the shared announcer where the page has one — follow whatever `wireSummaryEdit` uses), the editor collapses back to the badge + `Edit date` button; on non-2xx or network failure → the server `detail` (or the canned `Couldn't save the date — try again.` on a plain network error) into a `role="alert"` line (the phase-89 error-line idiom — the nearest existing error surface in this file), the input reverts to the stored date, the controls re-enable — the UI never claims a state the server didn't save. + - **No other surface:** the editor lives in `renderDocument`'s shared core only — the modal and the page both get it (both already call the core with `docAdminReady` available — verify `docAdminReady` is reachable in the modal's bundle context; `document-modal.js` imports `renderDocument` from this module, so the wiring rides along with the module — no second copy). +2. `frontend/assets/styles.css` — the editor's controls (the `.kb-summary-edit` / summary-editor rule family as the model, near it): `.doc-date-edit` (the button), the date input (sized, the global `:focus-visible` ring applies — no per-control rule, the phase-105 checkbox idiom), `:disabled` (opacity + `cursor: wait` — the `.git-source-remove:disabled` idiom), `role="alert"` line (the `.git-source-error` styling reuse or a local sibling), provenance comments citing phase 106 D7; contrast ≥4.5:1 verified + recorded in comments (house style). +3. `tests/unit/test_date_editor.py` (NEW — the read-the-assets-as-text pattern, task 08's suite extended or a sibling): + - `frontend/assets/document.js`: `wireDateEdit` exists and is called ONLY behind `docAdminReady()`'s `if (admin)` (a source-level pin — the string sequence `docAdminReady().then` … `wireDateEdit`); the PATCH URL is `/api/documents/date` (the single-source cross-file check — the endpoint string appears exactly once in the JS, matching `app/api/docs.py`'s route); the response-driven badge re-render (the `res.created_at` reference, NOT `input.value`); the revert link sends `date: null`; the disable-on-submit + revert-on-failure branches exist (the error-line `role="alert"` + the re-enable); the aria labels (`Edit creation date: `, `Document creation date`); + - `frontend/assets/styles.css`: the editor classes present with provenance comments. +4. Run `uv run pytest tests/unit/test_date_editor.py -q` + task 08's suite + the phase-57 suite's unit pins — green. + +## Testing & Quality +- Unit: the source-level wiring pins above (gate, endpoint, response-driven render, §7.4 branches, a11y strings). +- Coverage: **>90%** on `app/` (no `app/` code this task — the endpoint's coverage landed in task 05; the gate is the full-suite one). + +## Completion Criteria +- [ ] An admin sees an `Edit date` affordance next to the Created badge in BOTH the modal and the page (shared core — one implementation); the editor sets the date (input → `PATCH /api/documents/date` → the badge re-renders from the RESPONSE) and offers `Revert to sync` (→ `date: null`, the manual flag drops) +- [ ] The §7.4 lifecycle holds: controls disable on submit, a failure reverts the input to the stored value + announces in a `role="alert"` line + re-enables; the happy path announces through the live line after the badge update +- [ ] A non-admin / token holder / failed-whoami viewer is byte-for-byte the task-08 badge row (no button, no wiring, no extra request — the phase-57 split) +- [ ] `tests/unit/test_date_editor.py` passes; `uv run ruff check . && uv run pyright` clean diff --git a/.agents/phases/todo/106_document_dates/10_e2e_document_dates.md b/.agents/phases/todo/106_document_dates/10_e2e_document_dates.md new file mode 100644 index 0000000..d401c52 --- /dev/null +++ b/.agents/phases/todo/106_document_dates/10_e2e_document_dates.md @@ -0,0 +1,61 @@ +# Task 10 — E2E: `tests/e2e/test_document_dates.py` (isolation) + regressions + full gate + commit + +**Phase:** `106_document_dates` · **Source:** owner request 2026-09-13 — the whole item, proven end to end (dates sourced → stored → shown → editable → retrieval-weighted). + +## Objective +One dedicated Playwright suite proving the owner's item through the REAL page + REAL API + REAL importer (mock LLM — deterministic token-overlap embeddings, so the cosine/retrieval behavior is production-shaped; no git, no network — a local fixture dir with `os.utime`'d mtimes, built under `tmp_path_factory`, NEVER the shared `tests/fixtures/docs` whose 13-file counts are pinned by other suites). Then the phase's full gate and the single atomic commit. + +## Work +1. `tests/e2e/test_document_dates.py` (NEW) — module scaffolding from `tests/e2e/test_retrieval_quality.py` (`_import_fixtures`'s Settings-with-mock-port pattern, `_run_in_thread`, `_reset_db`, `e2e.auth_helpers.login`, the `app_url`/`mock_llm`/`db_ready` fixtures, the `source-chip` assertions) with the dedicated fixture tree (the module builds it ONCE per module under `tmp_path_factory`, `os.utime`'d — a `mkdocs + utime` helper at the top): + ``` + backups/retention.md utime 2020-01-01 03:04:06Z — THE CORRECT answer: + "The backup retention policy is 30 days; snapshots + are pruned nightly…" (rich in the question's tokens) + backups/retention-draft.md utime = now (default mtime) — the SIMILAR-but-wrong + doc: shares "backup retention policy" wording, + concludes "under review, no decision yet" + legacy/old-doc.md utime 2019-06-15 — single-doc folder (a clean + folder-`Updated` max: the 2019 date alone) + future/forward.md utime 2999-01-01 — the future-date case (→ today, D3) + ``` + Contract under test (docstring) — six tests, one per bullet: + 1. **`test_dates_landed_on_import`** — import the tree (real importer, mock LLM, in a thread): admin-cookie `GET /api/docs` — `retention.md`'s `created_at` ISO date-part = `2020-01-01`, `old-doc.md`'s = `2019-06-15`, `forward.md`'s = TODAY (the D3 future-fold, the test computes today in UTC); `GET /api/docs/tree` — file nodes carry the same dates; the `legacy` folder node's `updated_at` = the 2019 date (single-doc max), the source node's `updated_at` = the max of all (the `now`/today side); `indexed_at` on every row is UNCHANGED in meaning (still ≈ import time, after the created dates). + 2. **`test_file_and_folder_columns`** — real form login → the RAG view → the file table header order `… Chunks · Created · Indexed` (the `` sequence) and the drilled-in rows: `retention.md`'s Created cell `title` attribute = the ISO string (locale-stable — task 08's idiom) and its text contains `2020`; the folder table header order `Folder · Documents · Updated · Description`; at the top level the source row's `Updated` cell is non-empty; drilled into `legacy`'s parent, the `legacy` folder row's `Updated` cell `title` carries `2019-06-15`. + 3. **`test_viewer_shows_date_at_top`** — click `retention.md`'s row link (the real click — the same-page modal, phase 26): the modal's top meta row contains a badge with text starting `Created` whose `title` attribute = the 2020 ISO, and it DOM-precedes the `Indexed` badge (the date at the top of the clicked document, D8); the badge row also still shows `Indexed` + the source/format badges (no regression). + 4. **`test_old_correct_beats_new_similar`** — THE OWNER SCENARIO end to end: ask `How did I configure the backup retention policy?` → the grounded answer arrives (mock marker, no deflection), the FIRST `.source-chip` = `backups/retention.md` (the OLDER correct doc beats the newer similar one — the real retriever + the default recency boost over the mock's token-overlap embeddings); `query_log` — one row, `deflected is False`, `sources` contains `backups/retention.md`. (If the fixture wording doesn't produce the order under the DEFAULTS — the token-overlap geometry differs from task 07's axis vectors — adjust the FIXTURE TEXT until the old-correct doc is the clear top-1 (more exact question-phrase overlap in `retention.md`, the draft sharing only loose keywords), and record the working wording + the reason in the test docstring. Do NOT change the boost defaults here — task 07 owns them.) + 5. **`test_date_edit_and_sync_preserves`** — the admin-only edit through the REAL UI: open `legacy/old-doc.md` in the modal → the `Edit date` button is present (admin session) → click → set the date input to `2021-05-05` → Save → the badge re-renders from the response (title = a 2021 ISO) → admin-cookie `GET /api/docs` confirms `2021-05-05`. Re-run the import (in a thread, same tree — the mtimes are untouched): `old-doc.md` keeps `2021-05-05` (the manual flag, D1) while `retention.md` still reads 2020 (refreshed, not stale) and `forward.md` still reads today. Then the REVERT: open the editor again → `Revert to sync` → re-run the import → `old-doc.md`'s date is refreshed back to `2019-06-15` (the flag dropped — sync manages it again). + 6. **`test_anonymous_gate_and_editor_a11y`** — anonymous (no login): the RAG view shows the sign-in gate (no tables), a raw `PATCH /api/documents/date` with a date payload → 403; signed in (admin): the `Edit date` button's accessible name contains `legacy/old-doc.md` (the aria-label), the editor's date input has the `Document creation date` accessible name, is keyboard-reachable (Tab from the button), the status line is `role="status"` (and the error path's line `role="alert"` exists in the DOM — the phase-57/89 surfaces); the badge text pairs (text + formatting, never color alone — the monochrome-theme contract, B5). +2. **Regressions** — each in isolation (DB up), all green (task 06 changed pinned formats — the `ls` line, the `read` result, the `` block; task 04's unchanged-path date refresh must not move any content count): + - `uv run pytest tests/e2e/test_retrieval_quality.py -v --no-cov` (the fixture-import + ranking E2E — the mock-regex canary) + - `uv run pytest tests/e2e/test_whole_document_context.py -v --no-cov` (the `` block) + - `uv run pytest tests/e2e/test_agent_document_tools.py -v --no-cov` + - `uv run pytest tests/e2e/test_ls_tree_drilldown.py -v --no-cov` (the `ls` line format) + - `uv run pytest tests/e2e/test_read_truncation_cap.py -v --no-cov` (the `read` result shape) + - `uv run pytest tests/e2e/test_kb_tree.py -v --no-cov` + `uv run pytest tests/e2e/test_kb_tree_nav.py -v --no-cov` (the tree shape + the tables) + - `uv run pytest tests/e2e/test_document_viewer.py -v --no-cov` + `uv run pytest tests/e2e/test_edit_summaries.py -v --no-cov` (the viewer core + the sibling admin-edit idiom) + - `uv run pytest tests/e2e/test_import_documents.py -v --no-cov` + `uv run pytest tests/e2e/test_sync_button.py -v --no-cov` (importer counts + the sync detail) + - `uv run pytest tests/e2e/test_hidden_folders_toggle.py -v --no-cov` (phase 105 — the importer map idiom) + - `uv run pytest tests/e2e/test_smoke.py -v --no-cov` + (Where a suite pins a pre-phase format EXACTLY — an `ls` line without the date field, a `read` result without the `date:` line — update the pin in that test file to the phase-106 shape (mechanical, the new field is deterministic). A suite that breaks for any OTHER reason is a regression — fix the product code in its owning task's files, keep this phase's contract as written.) +3. **Full gate** (AGENTS.md rule 9 — non-negotiable): + - `uv run pytest` (unit + integration) green + - `uv run pytest --cov=app --cov-report=term-missing` — TOTAL **>90%** + - `uv run pytest tests/e2e/test_document_dates.py -v --no-cov` green in isolation + - `uv run ruff check . && uv run pyright` clean +4. **Commit** — one atomic Conventional-Commits commit, `--no-gpg-sign` (AGENTS.md rule 8), per the phase overview's Commit block: + ```bash + git add app/ alembic/versions/0020_documents_created_at.py scripts/ frontend/ tests/ .env.example .agents/phases/ && git commit --no-gpg-sign -m "feat(dates): document dates end to end — sourced at sync, shown in UI, editable, recency-weighted in retrieval" + ``` + (If the pipeline commits per task instead, fold everything into this phase's final commit and move the phase dir to `.agents/phases/complete/106_document_dates/` as the last action, per the pipeline gate.) + +## Testing & Quality +- This task IS the phase's E2E + gate + commit; no new application code (the E2E may reveal a one-line fix in an earlier task's work — fix it IN that task's file, keep the suite's contract as written). +- Coverage: **>90%** on `app/` (the validate.sh gate — enforced here, not assumed). + +## Completion Criteria +- [ ] `tests/e2e/test_document_dates.py` exists, maps 1:1 to the six contract bullets, and passes in isolation (`--no-cov`, DB up) +- [ ] The owner's scenario holds end to end: the older document that answers the question is the FIRST cited source over the newer similar one (defaults, real retriever, mock embeddings) +- [ ] The date edit round-trips through the real UI + API and SURVIVES a re-import (manual flag), and `Revert to sync` hands management back to the next import +- [ ] All ten regression suites pass in isolation (the format-pin updates are mechanical and live in the test files) +- [ ] The full gate is green: unit + integration, TOTAL coverage >90%, ruff + pyright clean +- [ ] One `--no-gpg-sign` commit contains the whole phase (app + alembic 0020 + scripts + frontend + tests + `.env.example` + the phase files) diff --git a/.agents/reports/106_document_dates/106_document_dates__01_created_at_column.a1.err b/.agents/reports/106_document_dates/106_document_dates__01_created_at_column.a1.err new file mode 100644 index 0000000..e69de29