brain-of-reese/.agents/phases/todo/106_document_dates/00_phase.md

Phase 106 — Document dates end to end: sourced at sync, displayed, editable, recency-weighted in retrieval

Source: Owner request 2026-09-13 (chat): documents "all have dates, either via their git timestamp or via file metadata (hopefully) preserved in the tar or zip archive process"; "After this phase, all documents must include the date when fed to the LLM"; "The UI must also show a date for every document at the top of that document when the user clicks it"; "If the date of a document can't be determined or is in the future then assume that document was created today"; "Date should be stored in the bor database and updated when sources are synced"; "Newer documents should be ranked higher in retrieval somehow, or at least given a boost, without breaking the existing retrieval process"; "last updated dates/timestamps on folders before the description column but after the documents column"; "For files, include a date/timestamp before the 'indexed' column"; "This timestamp should be editable so users can correct for errors"; "the timestamp can't be null so just set it to today's date during the migration and then, on sync, update the date"; "It's totally fine for a sync to cause a document or folder's date to get older."

Story: n/a (owner roadmap confirmation — the phase's E2E suite proves the item end to end).

Context: Every imported file becomes a documents row (app/models.py L101) with indexed_at (L112, the INDEX time — a different concept from this phase's document creation date, which it does not touch) and a nullable summary. The importer (app/rag/importer.py) walks each source root (iter_importable_files L171-194) and upserts per file in _index_file (L376-471): content-hash delta — added/updated (the row write, L405-421) or the early-return unchanged path (L401-404, no write today); import_sources (L213-340) already carries per-root maps keyed by str(root) — ignore_by_root (phase 89) and include_hidden_by_root (phase 105) — with the progress pre-walk and prune using the same per-root resolution; ImportSummary (L64-111) logs the greppable import: summary files=… line (PLAN §9). The two live entry points both resolve the git_sources rows, clone git rows through scripts/git_sync.clone_or_pull (app/api/sync.py::_run_sync per-row loop L233-263, import call L300-303; scripts/import_docs.py::_resolve_sources L168-226, clone L220, import call L331-334) and pass the per-root maps through — this phase adds a THIRD map (task 04). Verified git behavior (2026-09-13, scratch repos): git clone --depth 1 /local/path warns "--depth is ignored in local clones" and keeps FULL history, so local-path git sources can yield TRUE per-file last-commit dates; URL-transport clones (https/ssh/file://) are shallow, and in a shallow clone git log -1 --format=%cI -- <file> returns the TIP commit's date for EVERY existing file (the shallow boundary is each file's history root — a uniform per-repo date, real across repos, flat within one). git log --name-only --format=@@%cI walks newest-first and lists every working-tree file under the shallow tip. The archive unpacker (app/rag/archive_upload.py, phase 49) streams member content (_unpack_zip L214-238, _unpack_tar L241-270) but never restores member mtimes — extracted files carry the extraction time, so uploads today lose their dates (the owner's "hopefully" — this phase fixes it, task 03). Retrieval (A7): cosine top-100 ∪ FTS top-30 → RRF fuse() (k=60, app/rag/retriever.py L225-258) → select_documents (L540+, sorts chunks by fused score, dedupes parents, top-N=2 full texts, never truncated); the honesty gate and query_log.top_score are COSINE-based (untouched by a score-side boost); lexical candidates are detached Document rows rebuilt from two raw SQL projections (_LEXICAL_SQL L140-157, _NAME_HIT_SQL L277-300) that must gain the new column (task 06); the vector path returns ORM rows (the column comes for free). LLM surfaces: the HIGH prompt's <document> block (the pinned shape, app/rag/prompts.py L371-376: source/path/title attributes + full text), the read tool result (app/rag/agent.py L1173-1185: first line Document {source}/{path}: — the E2E mock's _READ_RESULT_PREFIX header contract, tests/e2e/mock_llm.py L867), and the ls file line (render_folder_listing L856-905: source: X | path: Y | title: Z, fed by _source_document_rows L658-669 + the pure group_folder_listing L707-780, cross-checked level-for-level against the UI tree builder in app/api/docs.py L326/L376). The E2E mock parses prompts/tool results with regexes that must stay compatible: _DOCUMENT_BLOCK_RE (L785-789 — breaks unless updated for the new attribute), _CATALOG_LINE_RE (L873-875 — title: .+$ tolerates an APPENDED | date: … field), _READ_RESULT_PREFIX (L867 — header line must stay byte-identical). UI: the RAG view's file table header Source | Path | Title | Chunks | Indexed (frontend/index.html L485-492, rows built by makeRow in frontend/assets/sources.js L1375-1410 — cell order [title, chunks, indexed_at] L1401) and the folder/source table Folder | Documents | Description (L470-478, rows makeSourceRow L1217-1235 / makeFolderRow L1237-1261); the view drills client-side over GET /api/docs/tree (phase 97/99 — pure builder build_kb_tree app/api/docs.py L377-490 over TreeDocRow tuples L296-302; file nodes carry indexed_at verbatim, folder nodes carry recursive documents counts); the document viewer (click → same-page modal, phase 26; full-page escape /document.html) renders through ONE shared core renderDocument (frontend/assets/document.js L118-176: title + the .doc-meta badge row — source · format · path · Indexed <date> · chunks — then content; fmtDate L85, metaBadge L97); the admin-only inline editor idiom is phase 57's summary editor (docAdminReady() gate, wireSummaryEdit, PATCH /api/documents/summary in app/api/docs.py L164-232 — embed-before-write, 404 unknown pair, admin gate). House patterns: Alembic head is 0019 (every migration ships a tested downgrade, A13); env settings are BOR_-prefixed Settings fields with the kill-switch validator pattern (agent_max_rounds: 0 disables, negative fails startup loudly, app/config.py L155-168); the manually_edited flag (folder_summaries, phase 97) is the precedent for owner corrections surviving the sync-time generator; integration tests seed real Postgres with deterministic axis vectors (tests/integration/test_name_hit_lexical.py — D=768 unit vectors, exact cosines); E2E story suites seed through the REAL importer against a fixture dir with the deterministic mock LLM (tests/e2e/test_retrieval_quality.py — _import_fixtures/_reset_db/source-chip assertions); tests/conftest.py holds the db fixture; tests/fakes.py the fake LLM.

Objective

Every indexed document carries a creation date from its source — the git last-commit date for git sources (true per-file for local-path checkouts, the repo tip-commit date for shallow URL checkouts), the file mtime for local directories and unpacked uploads (the unpackers now preserve member mtimes) — normalized (undetermined or future → today), stored NOT-NULL on documents.created_at (existing rows backfilled to the migration day per the owner), refreshed on every sync (allowed to go older), and editable by the admin (the correction survives syncs via a manual flag). The date is fed to the LLM on every document surface (the HIGH prompt's <document> block, the read result, the ls file line), shown in the UI at the top of a clicked document (viewer badge) and in both catalog tables (files: Created before Indexed; folders/sources: Updated after Documents, before Description — the subtree's max document date), and applied in retrieval as a small, env-tunable, kill-switchable recency boost on the RRF-fused score that NEVER lets a newer similar document outrank an older document that actually answers the question (pinned by a dedicated integration battery + the story E2E).

Dependencies

• 105_hidden_folders_toggle (complete) — the per-root str(root) map idiom in the importer both entry points already build (task 04 adds the third map alongside); its suites must stay green.
• 09_story_retrieval_quality (complete) — the A7 hybrid retriever, fuse()/select_documents, and the eval script the boost extends; the fixture-KB E2E pattern.
• 97_kb_tree_catalog / 99_kb_tree_table_and_back_nav (complete) — the pure tree builder + the RAG view tables/rows this phase's columns extend; the ls↔tree cross-check that a changed ls line must keep.
• 57_edit_document_summaries (complete) — the admin-only inline editor + PATCH /api/documents/… idiom the date editor copies.
• 28_git_based_sources (complete) — scripts/git_sync.py (the ONLY git-invocation site, A11) where the per-file date walk is added; --depth 1 stays (D10).
• 90_upload_no_scan / phase 49 uploads (complete) — the unpacker this phase makes mtime-preserving (task 03); the scan-deferred upload flow itself is untouched.

Design (shared by all tasks — the executor reads this, not the chat)

• Storage (task 01, D1). documents.created_at — DateTime(timezone=True) NOT NULL, server_default=func.now() — so every row of an existing deployment backfills to the migration moment (≈ today, the owner's instruction) and the next sync replaces it with the real sourced date (D4 makes the unchanged path write). documents.created_at_manual — Boolean NOT NULL, Python default=False, server_default=text("false") (the include_hidden L159-167 / manually_edited phase-97 column style). Alembic 0020_documents_created_at.py revises 0019; downgrade drops BOTH columns (A13).
• Normalization (task 02, D3). New module app/rag/doc_dates.py — pure, stdlib-only, the single choke point every date passes through:

  FUTURE_SKEW_TOLERANCE = timedelta(days=1)
  def normalize_doc_date(raw: datetime | None, now: datetime | None = None) -> datetime
  

  None (undetermined) → now (default datetime.now(UTC), date-part-irrelevant — full precision kept); naive raw → treated as UTC; aware raw → converted to UTC; raw > now + FUTURE_SKEW_TOLERANCE (a genuinely FUTURE date — the 1-day tolerance absorbs clock skew, pinned) → now. Also file_mtime_datetime(path: Path) -> datetime = datetime.fromtimestamp(path.stat().st_mtime, tz=UTC) (epoch mtimes are tz-agnostic — UTC is the correct rendering). No other module re-derives these rules.
• Source date extraction (task 03, D2/D10).
  • Archives — _unpack_zip: after each regular-file write, os.utime(dest, ns=(t, t)) from the member's DOS date_time (datetime(*member.date_time, tzinfo=timezone.utc).timestamp() — DOS mtimes are tz-agnostic like epochs); _unpack_tar: os.utime(dest, ns=(member.mtime, member.mtime)). Regular files ONLY (dirs/symlinks/hardlinks untouched — only regular files are ever indexed); the zip-bomb cap and every safety check are UNCHANGED (the utime calls come after _write_capped succeeds — a failed unpack still removes the partial tree).
  • Git — scripts/git_sync.py gains file_commit_dates(dest: Path) -> dict[str, datetime]: ONE git log --name-only --format=@@%cI walk through run_git (the A11 single-invocation site), parsed newest-first — per repo-relative POSIX path, the FIRST sighting wins (the latest commit that touched it); paths normalized to POSIX (/); ISO-strict parsed via datetime.fromisoformat. Fail-soft: any GitSyncError/parse failure → {} (logged) — the importer falls back to file mtimes; a date walk must never break a sync. Verified behavior: local-path checkouts (full history) → true per-file dates; shallow URL checkouts → the tip commit's date for every working-tree file (the shallow-boundary property, pinned by an integration test using a file:// clone).
• Importer semantics (task 04, D4). import_sources(…, doc_dates_by_root: dict[str, dict[str, datetime]] | None = None) — keyed by str(root) exactly like ignore_by_root/include_hidden_by_root; ONLY git roots are listed (the entry points build the map from file_commit_dates after the clone); unlisted roots (local dirs, uploads) take the mtime fallback. In _index_file (receives raw_date: datetime | None pre-resolved by the loop — map.get(rel) else None):
  • raw_date is None → file_mtime_datetime(full_path) (one stat).
  • added / updated (content change) → doc.created_at = normalize_doc_date(raw), doc.created_at_manual = False — a new content version is a new document date; a previous manual correction referred to the old content and is deliberately reset.
  • unchanged (hash match) → if doc.created_at_manual → leave the row entirely (the owner's correction survives syncs — the phase-97 manually_edited precedent); else recompute normalize_doc_date(raw) and, when it differs from the stored value, write + commit it (a date-only refresh) and count it in the NEW ImportSummary.dates_updated (additive field + the import: summary … dates_updated=N log term, PLAN §9). A date-only refresh is still unchanged: it NEVER counts toward added/updated/pruned → no sources_meta bump, no overview/folder-summary regeneration, no saved-chat staleness (D4 — content is the KB, the date is annotation).
  • Entry points: _run_sync — in the existing per-row loop, for kind=git rows after clone_or_pull: doc_dates_by_root[str(root)] = file_commit_dates(root); pass the map to import_sources; add "dates_updated": summary.dates_updated to the success detail (additive key). scripts/import_docs.py::_resolve_sources returns the map as a 4th tuple element (manual --source dirs and env-fallback rows contribute nothing); main passes it through (L331-334).
• API surface (task 05, D7/D8/D9).
  • GET /api/docs — DocSummary gains created_at: str (ISO-8601, doc.created_at.isoformat()) — the column joins the existing select + group_by (indexed_at L104/L107). GET /api/documents/content — DocContent gains created_at: str (L161 site).
  • PATCH /api/documents/date (NEW, admin-only, require_admin — the update_document_summary gate L164-232): body DateUpdate {source: str, path: str, date: str | None}. Unknown pair → 404 document not found (row-lookup semantics, no filesystem — the /documents/content rule). date present → datetime.fromisoformat (a bare YYYY-MM-DD or full ISO datetime; a malformed value → the 422 the model gives) → normalize_doc_date (D3 — a manually set future date also folds to today; consistency with the sourced path) → created_at set + created_at_manual = True. date null/absent → the CLEAR: created_at_manual = False only (the stored date stands until the next sync refreshes it — the API is DB-only and cannot re-read the source). Response DateResult {source, path, created_at: str, created_at_manual: bool} echoing the stored state. No LLM/embedding call (a date is never embedded — the phase-57 no-LLM contrast).
  • GET /api/docs/tree — TreeDocRow becomes a 6-tuple (source, path, title, chunks, indexed_at, created_at_iso); the endpoint's query adds Document.created_at (the indexed_at select/group_by site L543-546); KbTreeFile gains created_at: str; KbTreeFolder/KbTreeSource gain updated_at: str | None (D9: the subtree's MAX created_at, computed in the PURE builder as it recurses — None for a 0-document source, the summary-null shape); the builder threads the max through _source_node → _level_children (folders: the max of own files + child subfolders). No new table, no new column beyond task 01's (D9).
• LLM surfaces (task 06, D5).
  • Retriever plumbing: _LEXICAL_SQL + _NAME_HIT_SQL gain d.created_at AS created_at; the two detached Document(…) reconstructions (L360-372, L414-424) pass created_at=row.created_at. The vector path is ORM rows — nothing to do.
  • HIGH prompt (app/rag/prompts.py L371-376): the block becomes <document source="{doc.source}" path="{doc.path}" title="{doc.title}" date="{doc.created_at:%Y-%m-%d}"> — the UTC date part, attribute appended AFTER title (the only position; the attribute is always present — created_at is NOT NULL). No persona/teaching copy changes (phase 03 convention — the date rides existing locked text).
  • read tool (app/rag/agent.py L1173-1185): the first line stays Document {source}/{path}: BYTE-IDENTICAL (the mock's _READ_RESULT_PREFIX header contract — _read_results strips exactly that header to recover the path), the date is the SECOND line: Document {sp}:\ndate: {doc.created_at:%Y-%m-%d}\n{content…} — both the truncated (L1176) and the plain (L1183) results.
  • ls file line: APPEND | date: {YYYY-MM-DD} at the END — source: X | path: Y | title: Z | date: 2024-06-15 (the mock's _CATALOG_LINE_RE title: .+$ still matches — the appended field lands inside the greedy tail; do NOT insert before title, where the non-greedy path capture would swallow it). Plumbing: _source_document_rows returns (path, title, created_iso_date); group_folder_listing rows become Sequence[tuple[str, str, str]] and its file triples become (source, path, title, date); render_folder_listing renders the appended field; the NOT-A-FOLDER refusal path (L1121-1130) passes through unchanged in shape; the app/api/docs.py ls↔tree cross-check docstrings + tests/unit/test_kb_tree_builder.py cross-check tests compare the extended tuples (the "UI shows what the agent sees" invariant, extended with the date).
  • tests/e2e/mock_llm.py _DOCUMENT_BLOCK_RE (L785-789): make the date attribute an OPTIONAL group — r'title="[^"]*"(\sdate="[^"]*")?>' — so the mock tolerates pre- and post-phase shapes (house rule: marker/regex changes land with the prompt change, in this task).
• Recency boost (task 07, D6). Settings gains (the hybrid block, app/config.py L163-174): recency_boost: float = 0.001 (the max additive score for a zero-age document; 0 = off — pre-phase ranking byte-identical, the kill switch; negative → startup failure naming the field, the agent_max_rounds validator pattern) and recency_half_life_days: int = 365 (<= 0 fails startup). .env.example documents BOR_RECENCY_BOOST / BOR_RECENCY_HALF_LIFE_DAYS. app/rag/retriever.py gains the pure apply_recency_boost(chunks, *, now=None, weight=None, half_life_days=None) -> list[RetrievedChunk]: for each chunk, age_days = max(0.0, (now − doc.created_at).total_seconds() / 86400.0) (future clamps to 0 — a future-sourced doc reads as brand-new, consistent with D3), score += weight * math.exp(−age_days / half_life_days) (defaults from get_settings() when args omitted), then re-sorted with the EXISTING deterministic key (−score, −cosine, document.path, position) — with weight=0 the scores and order are untouched (pinned). retrieve() (L398-425) applies it AFTER fuse() when settings.recency_boost > 0 (the single apply site — chat API and scripts/eval_retrieval.py get it automatically; the script's printed table gains the document date + the post-boost effective score column). UNTOUCHED by design: the A8 honesty gate + query_log.top_score (cosine), weak_hit_titles (titles only), the never-truncated top-N contract, fuse() itself. The boost re-ranks which ≤N documents are FED (the whole point) but never truncates.
• The fine line (task 07's battery — the owner's warning, pinned permanently). Real-Postgres integration tests with axis vectors (the test_name_hit_lexical.py idiom — exact cosines, deterministic):
  1. Old-correct beats new-similar (THE owner scenario): doc A (created 2020-01-01) answers the question; doc B (created yesterday) is topically similar (shares tokens, weaker answer). Under DEFAULTS A must be select_documents()[0]; ALSO under recency_boost=0 (no-regression pin: relevance alone already ordered them).
  2. Boost is real: two near-twin documents (identical or 1-rank-apart fused scores, engineered so B's advantage is < the zero-age boost) — with defaults the NEWER ranks first; with weight=0 the OLDER ranks first (proving the boost is the differentiator, not drift).
  3. Decay: the boost halves at the half-life (pure-function unit pins: age 0 → full weight; age = half-life → weight/e… asserted as weight * exp(-1); age ≫ → ~0; future → full; weight=0 → byte-identical list order).
  4. If the defaults (0.001 / 365 d) fail test 1's margin, tune the DEFAULTS (not the test) until old-correct wins with ≥ a comfortable fused-score margin, and record the measured margin in the test docstring — the owner re-tunes live via BOR_RECENCY_BOOST.
• UI (tasks 08-09, D7/D8).
  • frontend/index.html: the file table header gains <th scope="col">Created</th> between Chunks and Indexed (L487-491); the folder table header gains <th scope="col">Updated</th> between Documents and Description (L473-475). No other shell change.
  • frontend/assets/sources.js: makeRow — insert the Created cell BEFORE the Indexed one (the L1401 value list becomes [title, chunks, fmtDate(d.created_at), fmtDate(d.indexed_at)]; the row object fed from tree file nodes (L1341-1350) carries created_at); makeSourceRow/makeFolderRow — one td with fmtDate(node.updated_at) (or "–" when null — the statLast null idiom L1294) between the count td and the description cell. All textContent (the XSS contract — never innerHTML with document-derived data).
  • frontend/assets/document.js (the ONE shared core — modal AND /document.html page): the .doc-meta badge row gains metaBadge("doc-created", \Created ${fmtDate(doc.created_at)}`)INSERTED BEFORE theIndexedbadge (L127) — the date at the top of a clicked document (D8). The badge class reuses thedoc-indexedstyling family (adoc-createdrule instyles.css` next to it, provenance comment citing phase 106; contrast ≥4.5:1 verified + recorded in the comment, house style).
  • Task 09 — the admin date editor (the phase-57 wireSummaryEdit idiom verbatim, gated by docAdminReady() — anonymous/token holders see the byte-identical phase-36/106 badge row, no button, no network call): an Edit date text button after the Created badge opens an inline editor in the meta row — a native <input type="date"> (prefilled doc.created_at's UTC date part, aria-label "Document creation date") + Save/Cancel buttons + a role="status" live line; Save → PATCH /api/documents/date {source, path, date: <input value>} — the endpoint is require_admin (the phase-57 split: the viewer content is require_user-gated, but the EDIT is admin-only; the editor is wired only for the admin, exactly as wireSummaryEdit); 200 → the badge re-renders from the response's created_at, announcer confirmation, status clears (§7.4); 4xx/5xx/network → the error line (role="alert"), the editor reverts to the stored value, controls re-enabled — the UI never claims a state the server didn't save. A clear affordance (the phase-57 "clear = empty" contrast): a "Revert to sync" link in the editor that sends {source, path, date: null} (D7's CLEAR — the manual flag drops; the date shows until the next sync).
• NOT touched: indexed_at (everywhere), the A7 fusion math / A8 gate / never-truncated contract, sources_meta staleness semantics, the clone strategy (--depth 1, D10), the upload scan-deferral (phase 90), suggestions, shared chats, the stat cards, AGENTS.md, .agents/PLAN.md, any completed phase. The ls top-level SOURCE lines and folder lines carry NO date (only FILE lines do — files are documents).

Tasks

1. 01_created_at_column.md — documents.created_at + documents.created_at_manual (model + alembic 0020) + default/round-trip/migration tests.
2. 02_date_normalization.md — app/rag/doc_dates.py (normalize_doc_date + file_mtime_datetime) + boundary unit tests.
3. 03_source_date_extraction.md — mtime-preserving zip/tar unpack + scripts/git_sync.py::file_commit_dates (local full-history vs shallow tip-date, fail-soft) + tests.
4. 04_importer_dates.md — import_sources date map + the added/updated/unchanged semantics + dates_updated + both entry-point wirings + unit & integration tests.
5. 05_date_apis.md — created_at on /api/docs + /api/documents/content, the admin PATCH /api/documents/date, and the tree's created_at/updated_at (schemas + pure builder + endpoint) + tests.
6. 06_llm_date_surfaces.md — retriever created_at plumbing, the <document date=…> block, the read date line, the ls appended date field, the mock-LLM regex, pin updates + tests.
7. 07_recency_boost.md — settings + validators, apply_recency_boost in retrieve(), eval_retrieval columns, .env.example, the fine-line integration battery + unit tests.
8. 08_ui_dates.md — the Created file column, the Updated folder/source column, the viewer Created badge (+ CSS/a11y) + source-level unit pins.
9. 09_date_editor.md — the admin-only inline date editor in the viewer (PATCH wiring, the clear/revert affordance, §7.4 lifecycle, a11y) + unit pins.
10. 10_e2e_document_dates.md — dedicated Playwright suite tests/e2e/test_document_dates.py (isolation) + regressions + full gate + atomic commit.

Testing & Quality

• Unit — tests/unit/test_doc_dates.py (task 02, the boundary matrix: None→today; naive-as-UTC; aware conversion; future just inside the 1-day tolerance keeps its date; future beyond → today; the mtime helper); tests/unit/test_archive_upload_dates.py (task 03 — a zip with an explicit date_time and a tar with an explicit mtime unpack to files carrying those mtimes; the cap/safety suite stays green); the file_commit_dates pins (task 03 — a scratch repo: local clone → per-file dates; a file:// shallow clone → tip date for every file; a missing .git/git failure → {} fail-soft); tests/unit/test_importer_dates.py (task 04 — a fake-LLM + tmp-tree: utime'd files land as created_at; the unchanged-refresh write + counter; the manual-flag skip; the content-change reset; doc_dates_by_root map hit vs mtime fallback); tests/unit/test_kb_tree_builder.py (extended, task 05 — file created_at verbatim, folder/source updated_at = subtree max, None for 0-doc sources, the extended cross-check tuples); tests/unit/test_prompts_dates.py (task 06 — the block attribute's position/format, the read line shape, the ls line's appended field, the weight-0 byte-identity of the boosted order lives in task 07's suite); tests/unit/test_retriever_recency.py (task 07 — the decay/weight/tie pins); tests/unit/test_sources_dates.py (tasks 08/09 — the house read-the-assets-as-text pattern: header cell ORDER pinned in both tables, the row cell order, the badge order in the shared core, the editor wiring + its aria labels + the single-source cross-file check that the editor PATCHes /api/documents/date).
• Integration — tests/integration/test_migration_0020.py (task 01, the house migration-suite pattern: upgrade from 0019 adds the columns with the server defaults — a pre-existing row reads created_at ≈ now() and created_at_manual is False; downgrade restores 0019's schema); tests/integration/test_git_file_dates.py (task 03 — real git scratch repos through clone_or_pull + file_commit_dates, both checkout kinds); tests/integration/test_importer_dates.py (task 04 — real Postgres: a backfilled-today row refreshed to its old utime on an unchanged re-import; the manual row untouched; the dates_updated counter; the date-only refresh does NOT bump sources_meta); tests/integration/test_docs_api_dates.py (task 05 — created_at in both reads; the PATCH matrix: set (round-trip + flag), malformed 422, null-clear (flag drops, date stands), 404 unknown pair, anonymous 403; the tree's updated_at on a nested fixture); tests/integration/test_agent_tools_dates.py (task 06 — the ls line + read result carry the date against real rows); tests/integration/test_recency_boost.py (task 07 — the fine-line battery, the owner's scenario, real Postgres + axis vectors); tests/e2e/test_document_dates.py (task 10, A16, isolation).
• E2E (mandatory, A16) — uv run pytest tests/e2e/test_document_dates.py -v --no-cov with the DB up.
• Coverage: >90% on app/ (uv run pytest --cov=app --cov-report=term-missing — the validate.sh gate; every new/modified module is fully covered by the suites above).

Completion Criteria

• Every documents row has a non-null created_at (fresh imports carry the sourced date — git last-commit or mtime; existing deployments' rows read the migration day until the first sync refreshes them); the 0020 migration upgrades from 0019 and downgrades cleanly on the dev/test DB.
• A sync (button OR CLI) refreshes dates: an unchanged file whose mtime moved gets the new date (dates_updated counts it); a file's date may go OLDER; a manually corrected date (created_at_manual) survives the sync while its siblings refresh; a content change resets the date and clears the manual flag; a date-only refresh leaves sources_meta.version, the overview, and the folder summaries untouched.
• A zip and a tar with old member timestamps unpack to files carrying those mtimes (the upload path's dates survive the archive process); a file with a future (beyond-skew) or missing date lands as today.
• The LLM sees the date on every document surface: the HIGH prompt's <document … date="YYYY-MM-DD"> block (mock-LLM regex updated, tolerant), the read result's date: second line (first line byte-identical), and the ls file line's appended | date: YYYY-MM-DD — with the ls↔tree cross-check still holding.
• The UI shows the date everywhere asked: the file table's Created column before Indexed; the folder/source table's Updated column after Documents, before Description (subtree max, – for empty); the clicked document's top meta row carries the Created <date> badge in BOTH the modal and the full page.
• An admin edits a document's date in the viewer (set + revert-to-sync), the change round-trips through PATCH /api/documents/date, and the next sync preserves the correction while refreshing the rest; non-admins see the byte-identical badge row with no editor; the failure path reverts + announces.
• Retrieval: the owner's scenario is pinned — an older document that answers the question ranks above a newer similar one that doesn't (defaults AND with the boost off); near-ties break toward the newer document (boost on) and toward the pre-phase order (boost off); BOR_RECENCY_BOOST=0 restores byte-identical ranking.
• uv run pytest green; uv run pytest --cov=app --cov-report=term-missing TOTAL >90%; uv run pytest tests/e2e/test_document_dates.py -v --no-cov green in isolation (DB up); the regression suites listed in task 10 green in isolation; uv run ruff check . && uv run pyright clean.
• One --no-gpg-sign commit; phase dir moved to .agents/phases/complete/ by the pipeline gate.

Locked decisions

• D1 — NOT-NULL storage + manual flag (owner-instructed 2026-09-13; flag = the phase-97 manually_edited precedent). documents.created_at NOT NULL, server-defaulted to now() at migration time (existing deployments get "today", corrected on the next sync — the owner's exact words); documents.created_at_manual (default false) protects an owner-corrected date from sync refreshes — owner corrections are never silently rewritten (the phase-97 decision, applied to the document date).
• D2 — Date provenance (owner-instructed). Git sources: the per-file LAST-COMMIT date (git log --name-only --format=@@%cI, committer date, one git call per source per sync, fail-soft to mtime). Everything else (local dirs, unpacked uploads): the file mtime — with the unpackers fixed to PRESERVE member mtimes (the owner's "hopefully" made real).
• D3 — Normalization (owner-instructed). Undetermined → today; future (beyond a 1-day clock-skew tolerance) → today; naive → UTC; all stored as UTC timestamptz at full precision (the UI formats). One choke point (app/rag/doc_dates.py).
• D4 — Sync semantics (owner-instructed). The date refreshes on every sync, including unchanged files, and may go OLDER (no monotonic guard). Added/updated content resets the date + clears the manual flag. A date-only refresh is unchanged for every existing gate (no sources_meta bump, no overview/summary regeneration, no staleness) and is counted in a NEW additive dates_updated counter (log line + sync detail).
• D5 — LLM surfaces (owner-instructed: "all documents must include the date when fed to the LLM"). The date="YYYY-MM-DD" attribute appended to the <document> block (after title); the read result's second line date: YYYY-MM-DD (first line byte-identical — the mock header contract); the ls FILE line's appended | date: YYYY-MM-DD (appended, never inserted — the mock line regex). No persona/teaching copy changes.
• D6 — The recency boost (owner-instructed: "ranked higher… or at least given a boost, without breaking the existing retrieval process"). An additive post-fusion term on the RRF score — fused + recency_boost·exp(−age_days/recency_half_life_days) — defaults 0.001 / 365 days, env-tunable (BOR_RECENCY_BOOST, BOR_RECENCY_HALF_LIFE_DAYS), 0 = byte-identical off, negatives fail startup loudly (the house validator pattern). Applied once in retrieve() after fuse(). The A7 math, the A8 cosine gate, query_log.top_score, and the never-truncated contract are untouched. The owner's old-correct-beats-new-similar scenario is a permanent integration pin.
• D7 — Editing (owner-instructed: "editable so users can correct for errors"). Admin-only PATCH /api/documents/date (the phase-57 split: the viewer stays user-gated, the edit is admin-gated); a date string sets + flags manual; null clears the flag (the date stands until the next sync). The editor lives in the shared viewer core (modal + page), phase-57 idiom, §7.4 never-stale lifecycle.
• D8 — UI columns/badge (owner-instructed positions, verbatim). Files: Created between Chunks and Indexed. Folders/sources: Updated between Documents and Description. Clicked document: the Created badge in the top meta row, before Indexed.
• D9 — Folder dates are derived. No folder date storage: updated_at = the subtree's max document created_at, computed in the pure tree builder (one concept end to end — the same recursion that counts documents); None for an empty source.
• D10 — No clone-strategy change (phase 28 stands). --depth 1 stays. Consequence (verified 2026-09-13): local-path git sources clone with FULL history → true per-file dates; URL git sources are shallow → every file carries the repo's TIP-commit date (uniform within the repo — no intra-repo distortion, a real cross-source signal, refreshed on every pull). Revisit only if the owner later wants intra-repo recency on URL sources.

A7 note (PLAN anchor, owner-permitted extension 2026-09-13): the recency boost is an additive POST-fusion re-ranking on A7's fused score, requested by the owner in this phase's ask ("newer documents should be ranked higher in retrieval… without breaking the existing retrieval process"). A7's hybrid mechanics, windows, k, and never-truncated contract are unchanged; when the owner signs off, PLAN.md §2's A7 row gains a dated revision note (the skill that authors phases never edits PLAN.md).

Commit

git add app/ alembic/versions/0020_documents_created_at.py scripts/ frontend/ tests/ .env.example .agents/phases/ && git commit --no-gpg-sign -m "feat(dates): document dates end to end — sourced at sync, shown in UI, editable, recency-weighted in retrieval"