phase: 106_document_dates

Everything is verified green. Final report:

**Phase 106 — Document dates (final verification pass; all 10 tasks already complete)**

- Verified all phase artifacts: alembic `0020` (dev DB at `0020`), `doc_dates.py`, git `file_commit_dates`, importer `doc_dates_by_root`/`dates_updated`, both entry-point wirings, date APIs + tree `created_at`/`updated_at`, LLM surfaces (prompt block, `read` line 2, appended `ls` field), `apply_recency_boost` in `retrieve()`, UI columns/badge, admin editor, mock-LLM regex — all present and correct; no defects found, no fixes needed.
- `uv run pytest --cov=app --cov-report=term-missing` → **2299 passed, TOTAL 99%** (>90% ✓)
- `uv run pytest tests/e2e/test_document_dates.py -v --no-cov` → **6/6 passed** in isolation (DB up)
- 12 regression E2E suites (retrieval_quality, whole_document_context, agent_document_tools, ls_tree_drilldown, read_truncation_cap, kb_tree, kb_tree_nav, document_viewer, edit_summaries, import_documents, sync_button, hidden_folders_toggle, smoke) → **all green in isolation**
- `uv run ruff check .` → clean; `uv run pyright` → **0 errors, 0 warnings**

**Completion criteria:** 1) non-null `created_at` + 0020 upgrade/downgrade on dev DB ✓ (real-Alembic integration tests) 2) sync refresh/older/manual-persists/content-reset/no sources_meta bump ✓ 3) zip/tar mtime + future→today ✓ 4) LLM date surfaces + cross-check ✓ 5) UI Created/Updated/badge positions ✓ 6) admin editor set+revert round-trip ✓ 7) old-correct-beats-new-similar (defaults & boost-off) + near-tie + `BOR_RECENCY_BOOST=0` byte-identical ✓ 8) full gate ✓ 9) commit/phase-move — left to harness per instructions.

- **Notable:** recency default tuned 0.001 → **0.0007** (task 07 step 5 explicitly permits; measured margins recorded in `test_recency_boost.py` docstring).
- **Next pending phase:** none — `todo/` holds only this phase.

tests/integration/test_migration_0020.py

@@ -0,0 +1,359 @@
+"""Integration: migration 0020 (documents.created_at / created_at_manual)
+schema contract (phase 106, task 01).
+
+Drives the **real Alembic engine** against the live dev database
+(``podman compose up -d db``), mirroring the house pattern of
+``test_migration_0019.py`` (information_schema assertions on the state the
+migration must leave). The tests target the 0019 → 0020 step explicitly so
+later migrations cannot break the pins:
+
+* upgrade 0019 → 0020 → both columns exist with the full contract —
+  ``created_at`` TIMESTAMP WITH TIME ZONE NOT NULL, server default
+  ``now()``; ``created_at_manual`` BOOLEAN NOT NULL, server default
+  ``false`` — while the 0019 ``documents`` schema (incl. ``indexed_at``,
+  ``summary``) survives;
+* a ``documents`` row inserted while the DB is at 0019 backfills
+  ``created_at ≈ now()`` (the D1 backfill-to-today, within a few seconds
+  of the upgrade moment) and ``created_at_manual is False``; a row written
+  after the upgrade without the columns takes both server defaults;
+* the ORM contract agrees: a freshly inserted ``Document`` (nothing passed)
+  reads ``created_at_manual is False`` + non-null ``created_at``, and an
+  explicit ``created_at`` + ``created_at_manual=True`` round-trips through
+  a fresh session;
+* downgrade to 0019 → both columns GONE (A13) while the row + its content
+  survive; upgrade back to 0020 → both columns back (round-trip).
+
+The ``alembic`` fixture guarantees the DB ends at head even if a test
+fails or the process is interrupted.
+"""
+from __future__ import annotations
+
+import hashlib
+import uuid
+from collections.abc import Iterator
+from datetime import UTC, datetime, timedelta
+from typing import Any
+
+import pytest
+from alembic.config import Config
+from sqlalchemy import text
+from sqlalchemy.orm import Session
+
+from alembic import command
+from app.db import SessionLocal, db_available
+from app.models import Document
+
+SOURCE = "mig0020"
+CONTENT = "content here"
+CONTENT_HASH = hashlib.sha256(CONTENT.encode()).hexdigest()
+# The backfill is evaluated by the ALTER at the upgrade moment; the 5 s
+# slack each side absorbs test-process scheduling without weakening the
+# "≈ now()" pin (the DB and the test share the host clock).
+SLACK = timedelta(seconds=5)
+
+
+@pytest.fixture()
+def alembic(db: Session) -> Iterator[Config]:
+    """Real Alembic config bound to the dev DB (URL from app settings).
+
+    Starts at head (repairs an interrupted earlier run); teardown upgrades
+    to head no matter what happened, so the dev DB is never left below
+    head.
+    """
+    if not db_available():
+        pytest.skip("Postgres not reachable — run `podman compose up -d db` first")
+    cfg = Config()  # no alembic.ini file — env.py gets the URL from app config
+    cfg.set_main_option("script_location", "alembic")
+    command.upgrade(cfg, "head")
+    try:
+        yield cfg
+    finally:
+        # Release the test session's open transaction BEFORE the repair
+        # DDL: an idle-in-transaction SELECT holds an ACCESS SHARE lock
+        # on ``documents``, which would deadlock the repair's
+        # ``ALTER TABLE`` (0020) forever.
+        db.rollback()
+        command.upgrade(cfg, "head")
+
+
+def _version(db: Session) -> str | None:
+    return db.execute(text("SELECT version_num FROM alembic_version")).scalar()
+
+
+def _column(db: Session, column: str) -> tuple[Any, ...] | None:
+    """(data_type, is_nullable, column_default) for one documents column."""
+    row = db.execute(
+        text(
+            "SELECT data_type, is_nullable, column_default"
+            " FROM information_schema.columns"
+            " WHERE table_name = 'documents' AND column_name = :c"
+        ),
+        {"c": column},
+    ).fetchone()
+    return tuple(row) if row is not None else None
+
+
+def _row_dates(db: Session, path: str) -> tuple[Any, Any]:
+    """(created_at, created_at_manual) for one documents row."""
+    row = db.execute(
+        text("SELECT created_at, created_at_manual FROM documents WHERE path = :p"),
+        {"p": path},
+    ).fetchone()
+    assert row is not None, f"documents row {path!r} must exist"
+    return row[0], row[1]
+
+
+def _insert_sql(db: Session, path: str) -> uuid.UUID:
+    """Insert one documents row (the pre-0020 column shape — the new
+    columns, when present, are omitted so the server defaults apply)."""
+    doc_id = uuid.uuid4()
+    db.execute(
+        text(
+            "INSERT INTO documents (id, source, path, full_path, title, content,"
+            " content_hash) VALUES (:id, :s, :p, :fp, :t, :c, :h)"
+        ),
+        {
+            "id": doc_id,
+            "s": SOURCE,
+            "p": path,
+            "fp": f"/tmp/{path}",
+            "t": f"Doc {path}",
+            "c": CONTENT,
+            "h": CONTENT_HASH,
+        },
+    )
+    db.commit()
+    return doc_id
+
+
+def _delete_by_path(db: Session, path: str) -> None:
+    db.execute(text("DELETE FROM documents WHERE path = :p"), {"p": path})
+    db.commit()
+
+
+def test_upgrade_to_0020_adds_created_at(db: Session, alembic: Config) -> None:
+    """Upgrade 0019 → 0020: both columns exist with the full contract
+    (``created_at`` TIMESTAMP WITH TIME ZONE NOT NULL default ``now()``;
+    ``created_at_manual`` BOOLEAN NOT NULL default ``false``), are ABSENT
+    at 0019, a pre-0020 row backfills ``created_at ≈ now()`` + the
+    manual flag ``false`` (D1), a new row without the columns takes both
+    server defaults, and an explicit ``true`` round-trips — while the
+    0019 table contract (``indexed_at``, ``summary``) survives."""
+    command.downgrade(alembic, "0019")  # start from the pre-0020 state
+    assert _version(db) == "0019"
+    assert _column(db, "created_at") is None, "created_at must be absent at 0019"
+    assert _column(db, "created_at_manual") is None, (
+        "created_at_manual must be absent at 0019"
+    )
+
+    path_pre = "pre-existing.md"
+    _insert_sql(db, path_pre)  # no created_at* columns exist at 0019
+    try:
+        window_start = datetime.now(UTC)
+        command.upgrade(alembic, "0020")
+        window_end = datetime.now(UTC)
+        assert _version(db) == "0020", "alembic_version must be at 0020"
+
+        created = _column(db, "created_at")
+        assert created is not None, "documents.created_at is missing"
+        assert created[0] == "timestamp with time zone", (
+            "created_at must be TIMESTAMP WITH TIME ZONE"
+        )
+        assert created[1] == "NO", "created_at must be NOT NULL"
+        assert created[2] is not None and "now()" in str(created[2]), (
+            "created_at must carry the `now()` server default"
+        )
+
+        manual = _column(db, "created_at_manual")
+        assert manual is not None, "documents.created_at_manual is missing"
+        assert manual[0] == "boolean", "created_at_manual must be BOOLEAN"
+        assert manual[1] == "NO", "created_at_manual must be NOT NULL"
+        assert manual[2] is not None and "false" in str(manual[2]), (
+            "created_at_manual must carry the `false` server default"
+        )
+
+        # The pre-0020 row backfilled created_at ≈ now() (D1 — the owner's
+        # "set it to today's date during the migration") + flag false.
+        backfilled, manual_pre = _row_dates(db, path_pre)
+        assert backfilled is not None, "the backfilled created_at must be non-null"
+        assert backfilled.tzinfo is not None, "created_at must be tz-aware"
+        backfilled_utc = backfilled.astimezone(UTC)
+        assert window_start - SLACK <= backfilled_utc <= window_end + SLACK, (
+            f"the backfill must be ≈ the upgrade moment (got {backfilled_utc})"
+        )
+        assert manual_pre is False, (
+            "the backfilled row must read created_at_manual is False"
+        )
+
+        # A row written without the columns takes both server defaults.
+        path_new = "new-row.md"
+        _insert_sql(db, path_new)
+        try:
+            created_new, manual_new = _row_dates(db, path_new)
+            assert created_new is not None
+            created_new_utc = created_new.astimezone(UTC)
+            assert window_end - SLACK <= created_new_utc <= datetime.now(UTC) + SLACK, (
+                f"an omitted created_at takes the `now()` server default"
+                f" (got {created_new_utc})"
+            )
+            assert manual_new is False, (
+                "an omitted flag takes the `false` server default"
+            )
+
+            # The flag round-trips through an explicit ``true``.
+            db.execute(
+                text("UPDATE documents SET created_at_manual = true WHERE path = :p"),
+                {"p": path_new},
+            )
+            db.commit()
+            assert _row_dates(db, path_new)[1] is True, (
+                "created_at_manual = true must round-trip"
+            )
+        finally:
+            _delete_by_path(db, path_new)
+
+        # The 0019 schema survives the additive upgrade.
+        indexed = _column(db, "indexed_at")
+        assert indexed is not None, "documents.indexed_at (0001) must survive the upgrade"
+        assert indexed[0] == "timestamp with time zone" and indexed[1] == "NO", (
+            "documents.indexed_at (0001) must keep its 0019 contract after the upgrade"
+        )
+        summary = _column(db, "summary")
+        assert summary is not None and summary[0] == "text" and summary[1] == "YES", (
+            "documents.summary (0004) must survive the upgrade"
+        )
+    finally:
+        _delete_by_path(db, path_pre)
+
+
+def test_orm_fresh_row_defaults_and_explicit_round_trips(
+    db: Session, alembic: Config
+) -> None:
+    """The ORM contract agrees with the column contract: a freshly
+    inserted ``Document`` (nothing passed for the new columns) reads
+    ``created_at_manual is False`` + non-null ``created_at`` (the server
+    default took effect — D1), and an explicit ``created_at`` +
+    ``created_at_manual=True`` round-trips through a fresh session."""
+    command.upgrade(alembic, "head")
+    path_default = "orm-default.md"
+    path_explicit = "orm-explicit.md"
+    try:
+        # Fresh row, both new columns omitted → server/Python defaults.
+        row_default = Document(
+            source=SOURCE,
+            path=path_default,
+            full_path=f"/tmp/{path_default}",
+            title="Default",
+            content=CONTENT,
+            content_hash=CONTENT_HASH,
+        )
+        db.add(row_default)
+        db.commit()
+        db.expire_all()
+        reloaded_default = db.get(Document, row_default.id)
+        assert reloaded_default is not None, "the fresh row must be readable"
+        assert reloaded_default.created_at is not None, (
+            "a fresh row must read a non-null created_at (server default)"
+        )
+        assert reloaded_default.created_at_manual is False, (
+            "a fresh row must read created_at_manual is False"
+        )
+
+        # Explicit created_at + created_at_manual=True round-trip through
+        # a FRESH session.
+        explicit = datetime(2020, 6, 15, 12, 30, 45, 123456, tzinfo=UTC)
+        row_explicit = Document(
+            source=SOURCE,
+            path=path_explicit,
+            full_path=f"/tmp/{path_explicit}",
+            title="Explicit",
+            content=CONTENT,
+            content_hash=CONTENT_HASH,
+            created_at=explicit,
+            created_at_manual=True,
+        )
+        db.add(row_explicit)
+        db.commit()
+        with SessionLocal() as fresh:
+            reloaded = fresh.get(Document, row_explicit.id)
+            assert reloaded is not None, "the row must exist in a fresh session"
+            assert reloaded.created_at is not None
+            assert reloaded.created_at.astimezone(UTC) == explicit, (
+                "the explicit created_at must round-trip through the DB"
+            )
+            assert reloaded.created_at_manual is True, (
+                "created_at_manual=True must round-trip through the DB"
+            )
+    finally:
+        _delete_by_path(db, path_default)
+        _delete_by_path(db, path_explicit)
+
+
+def test_downgrade_to_0019_drops_the_columns(db: Session, alembic: Config) -> None:
+    """Downgrade 0020 → 0019: both columns are gone (A13 — fully
+    reversible) while the row + its content survive, and the rest of the
+    0019 table contract (``indexed_at``) is intact."""
+    command.upgrade(alembic, "head")
+    path = "survivor.md"
+    _insert_sql(db, path)
+    try:
+        command.downgrade(alembic, "0019")
+        assert _version(db) == "0019"
+        assert _column(db, "created_at") is None, "created_at must be dropped"
+        assert _column(db, "created_at_manual") is None, (
+            "created_at_manual must be dropped"
+        )
+        row = db.execute(
+            text(
+                "SELECT path, title, content, content_hash, indexed_at"
+                " FROM documents WHERE path = :p"
+            ),
+            {"p": path},
+        ).fetchone()
+        assert row is not None and row[0] == path, (
+            "the row must survive the column drops"
+        )
+        assert row[1] == "Doc survivor.md" and row[2] == CONTENT, (
+            "title + content must survive the column drops"
+        )
+        assert row[3] == CONTENT_HASH, "the content hash must survive the drop"
+        assert row[4] is not None, "indexed_at must survive the column drops"
+
+        indexed = _column(db, "indexed_at")
+        assert indexed is not None and indexed[0] == "timestamp with time zone", (
+            "documents.indexed_at must survive the downgrade"
+        )
+    finally:
+        _delete_by_path(db, path)
+    # Repair: the fixture teardown re-upgrades to head.
+
+
+def test_upgrade_round_trip_restores_the_columns(db: Session, alembic: Config) -> None:
+    """Downgrade to 0019, then upgrade back to 0020: both columns are
+    back with the full contract (TIMESTAMPTZ NOT NULL default ``now()``;
+    BOOLEAN NOT NULL default ``false``)."""
+    command.downgrade(alembic, "0019")
+    command.upgrade(alembic, "0020")
+    assert _version(db) == "0020", "round-trip upgrade must land at 0020"
+
+    created = _column(db, "created_at")
+    assert created is not None, "documents.created_at must be back"
+    assert created[0] == "timestamp with time zone", (
+        "created_at must be TIMESTAMP WITH TIME ZONE after the round-trip"
+    )
+    assert created[1] == "NO", "created_at must be NOT NULL after the round-trip"
+    assert created[2] is not None and "now()" in str(created[2]), (
+        "the `now()` server default must survive the round-trip"
+    )
+
+    manual = _column(db, "created_at_manual")
+    assert manual is not None, "documents.created_at_manual must be back"
+    assert manual[0] == "boolean", (
+        "created_at_manual must be BOOLEAN after the round-trip"
+    )
+    assert manual[1] == "NO", (
+        "created_at_manual must be NOT NULL after the round-trip"
+    )
+    assert manual[2] is not None and "false" in str(manual[2]), (
+        "the `false` server default must survive the round-trip"
+    )