Files
brain-of-reese/tests/integration/test_migration_0006.py
T
ducoterra f20d40e4fe
Build and Push Containers / build-and-push-app (push) Successful in 1m44s
Build and Push Containers / build-and-push-db (push) Successful in 12s
phase: 188_chunks_hnsw_index
**Phase 188 — final verification pass: all criteria green.**

- **Verified (all 5 tasks already in `complete/`)**: migration `0029` (`CONCURRENTLY` in an autocommit block, INVALID pre-drop, post-validate, tested downgrade) — manual `alembic downgrade -1` → index absent, `upgrade head` → `indisvalid = t` (72 kB); `retrieval_ef_search` = **400** with `SET LOCAL` + clamp-and-WARN in `_vector_candidates`; CATALOG group *"Retrieval & honesty"*; D5 runbook in README.
- `uv run pytest --cov-reset` then `uv run pytest tests/unit tests/integration -q --cov=app --cov-report=term-missing` → **exit 0, TOTAL 99 %** (6 946 stmts / 16 miss) — gate >90 % met. (A stale `.coverage` left by an earlier session produced a spurious 90.00 %/3-failure reading; `--cov-reset` cleared it.)
- `uv run pytest tests/e2e/test_vector_index_recall.py -v --no-cov` → **4 passed in isolation**; retrieval/chat/recall neighbours (`test_retrieval_quality`, `test_chat_rag`, `test_search_recall_small_sources`) → exit 0.
- `uv run ruff check .` → **All checks passed**; `uv run pyright` (whole repo) → **0 errors**.
- **Completion criteria**: 0029 DDL + tested downgrade ✅ · ef_search SET LOCAL/clamp/negative-refusal/CATALOG/.env.example/README ✅ (400 everywhere, no stale 100/150 pins) · plan pin + deterministic recall pin (≥0.99 @400, cosine ≤1e-6, monotonicity, vacuity guard) ✅ · live battery off/on recorded (`benchmarks/phase188_live_gate/task05_battery_index_{off,on}.txt` byte-identical, `deterministic_gate_recall.txt`, build 161 s / 1 625 MB / 81 MB RSS, phase report `.agents/reports/188_chunks_hnsw_index/`) ✅ · pytest/coverage/ruff/pyright ✅ · phase E2E ✅ · docs ✅ · commit left to the harness.
- **Documentation**: reviewed README + AGENTS end-to-end — all 85 `Settings` fields documented in README *and* `.env.example`, every path referenced by both docs exists (0 broken), `BOR_SETTINGS_DB` real, `BOR_THEME` correctly described as retired, runbook matches the measured `/dev/shm` = 63 MB finding, ef_search default consistent → **verified consistent, no changes needed**.
- **Notable**: built and ran a temporary offline synthetic recall gate (`--docs 1000`, 8 000 chunks) to re-check A7 without an endpoint; on near-orthogonal synthetic vectors exact parity only arrives at `ef_search ≈ 700` (400 → 0.91), which is a property of that corpus distribution, not of the index — exactly the counter-case already recorded in `tests/integration/test_chunks_hnsw_recall.py`. Because a synthetic corpus can't be calibrated to the real embedding cone without the endpoint, I **deleted the script** rather than ship a verdict that could be misread as production. `AIPI_KEY` is absent in this environment (and `100.117.2.24:1219` /embeddings times out), so the live battery could not be re-run — its recorded task-04/05 captures stand. Scratch DB `brain_of_reese_hnswgate` dropped; repo left clean.
- **Next pending phase**: `189_chat_save_long_turns`.
2026-10-17 03:18:23 -04:00

201 lines
7.7 KiB
Python

"""Integration: migration 0006 (git_sources) schema contract.
Drives the **real Alembic engine** against the live dev database
(``podman compose up -d db``), mirroring the style of
``test_migration_0004.py`` / ``test_migration_0005.py`` (information_schema
assertions on the state the migration must leave). The tests target
revision ``0006`` explicitly so later migrations (0007, …) cannot break
them:
* upgrade 0005 → 0006 → ``git_sources`` exists with exactly the three
columns the phase locks in (``id UUID PK``, ``url TEXT NOT NULL``,
``added_at TIMESTAMPTZ NOT NULL`` default ``now()``) plus the unique
index ``uq_git_sources_url`` (duplicate URLs rejected with an
IntegrityError);
* an insert without ``added_at`` gets the server-stamped default;
* downgrade to 0005 → the table (and its index) is gone;
* upgrade back to 0006 → it is back (round-trip).
The ``alembic`` fixture guarantees the DB ends at head even if a test
fails or the process is interrupted.
"""
from __future__ import annotations
from collections.abc import Iterator
from typing import Any
import pytest
from alembic.config import Config
from sqlalchemy import text
from sqlalchemy.exc import IntegrityError
from sqlalchemy.orm import Session
from alembic import command
from app.db import db_available
@pytest.fixture()
def alembic(db: Session) -> Iterator[Config]:
"""Real Alembic config bound to the dev DB (URL from app settings).
Starts at head (repairs an interrupted earlier run); teardown upgrades
to head no matter what happened, so the dev DB is never left below
head.
"""
if not db_available():
pytest.skip("Postgres not reachable — run `podman compose up -d db` first")
cfg = Config() # no alembic.ini file — env.py gets the URL from app config
cfg.set_main_option("script_location", "alembic")
command.upgrade(cfg, "head")
try:
yield cfg
finally:
# Release the test session's open transaction BEFORE the repair
# DDL: psycopg3 keeps its snapshot alive while idle-in-transaction,
# and 0029's CREATE INDEX CONCURRENTLY phase 3 waits for every
# older-snapshot transaction — an unreleased session would park
# the build forever.
db.rollback()
command.upgrade(cfg, "head")
def _column(db: Session, column: str) -> tuple[Any, ...] | None:
"""(data_type, is_nullable, column_default) for one git_sources column."""
row = db.execute(
text(
"SELECT data_type, is_nullable, column_default"
" FROM information_schema.columns"
" WHERE table_name = 'git_sources' AND column_name = :c"
),
{"c": column},
).fetchone()
return tuple(row) if row is not None else None
def _version(db: Session) -> str | None:
return db.execute(text("SELECT version_num FROM alembic_version")).scalar()
def _unique_url_index(db: Session) -> int:
"""1 iff ``uq_git_sources_url`` exists as a UNIQUE index."""
count: Any = db.execute(
text(
"SELECT count(*) FROM pg_indexes"
" WHERE tablename = 'git_sources' AND indexname = 'uq_git_sources_url'"
" AND indexdef ILIKE 'CREATE UNIQUE%'"
)
).scalar()
assert count is not None, "pg_indexes count must be an int"
return int(count)
def _insert_url(db: Session, url: str) -> None:
db.execute(
text("INSERT INTO git_sources (id, url) VALUES (gen_random_uuid(), :u)"),
{"u": url},
)
db.commit()
def test_upgrade_to_0006_creates_git_sources(db: Session, alembic: Config) -> None:
"""Upgrade 0005 → 0006: the table exists with the locked column types,
nullability, server default, primary key, and unique URL index."""
command.downgrade(alembic, "0005") # start from the pre-0006 state
assert _version(db) == "0005"
command.upgrade(alembic, "0006")
assert _version(db) == "0006", "alembic_version must be at 0006"
pk = db.execute(
text(
"SELECT column_name FROM information_schema.table_constraints tc"
" JOIN information_schema.key_column_usage kcu"
" ON tc.constraint_name = kcu.constraint_name"
" WHERE tc.table_name = 'git_sources' AND tc.constraint_type = 'PRIMARY KEY'"
)
).scalar()
assert pk == "id", "git_sources primary key must be id"
id_col = _column(db, "id")
assert id_col is not None, "git_sources.id is missing"
assert id_col[0] == "uuid", "git_sources.id must be UUID"
assert id_col[1] == "NO", "git_sources.id must be NOT NULL"
url = _column(db, "url")
assert url is not None, "git_sources.url is missing"
assert url[0] == "text", "git_sources.url must be TEXT"
assert url[1] == "NO", "git_sources.url must be NOT NULL"
added = _column(db, "added_at")
assert added is not None, "git_sources.added_at is missing"
assert added[0] == "timestamp with time zone", "git_sources.added_at must be TIMESTAMPTZ"
assert added[1] == "NO", "git_sources.added_at must be NOT NULL"
assert added[2] is not None and "now()" in added[2], (
"git_sources.added_at must have server default now()"
)
assert _unique_url_index(db) == 1, "uq_git_sources_url unique index is missing"
def test_added_at_defaults_to_now(db: Session, alembic: Config) -> None:
"""An insert without ``added_at`` gets the server-stamped default —
the API layer never sets it itself (phase 35 task 02)."""
command.upgrade(alembic, "head")
try:
_insert_url(db, "https://git.example.com/mig-test/added-at.git")
stamped = db.execute(
text(
"SELECT added_at IS NOT NULL FROM git_sources"
" WHERE url = 'https://git.example.com/mig-test/added-at.git'"
)
).scalar()
assert stamped is True, "git_sources.added_at must be stamped by the server"
finally:
db.execute(
text(
"DELETE FROM git_sources"
" WHERE url = 'https://git.example.com/mig-test/added-at.git'"
)
)
db.commit()
def test_duplicate_url_rejected(db: Session, alembic: Config) -> None:
"""The unique index is what the API's 409 relies on: a duplicate URL
raises IntegrityError."""
command.upgrade(alembic, "head")
try:
_insert_url(db, "https://git.example.com/mig-test/dup.git")
with pytest.raises(IntegrityError):
_insert_url(db, "https://git.example.com/mig-test/dup.git")
finally:
db.rollback() # the IntegrityError aborts the open transaction
db.execute(
text("DELETE FROM git_sources WHERE url = 'https://git.example.com/mig-test/dup.git'")
)
db.commit()
def test_downgrade_to_0005_drops_table(db: Session, alembic: Config) -> None:
"""Downgrade to 0005: the table (and its index) is dropped (A13 —
reversible)."""
command.downgrade(alembic, "0005")
assert _version(db) == "0005"
exists = db.execute(text("SELECT to_regclass('public.git_sources') IS NOT NULL")).scalar()
assert exists is False, "git_sources must be dropped by the downgrade"
def test_upgrade_round_trip_restores_table(db: Session, alembic: Config) -> None:
"""Downgrade to 0005, then upgrade back to 0006: table, defaults, and
the unique index are back."""
command.downgrade(alembic, "0005")
command.upgrade(alembic, "0006")
assert _version(db) == "0006", "round-trip upgrade must land at 0006"
added = _column(db, "added_at")
assert added is not None and added[2] is not None and "now()" in added[2], (
"git_sources.added_at must keep its now() default after the round-trip"
)
assert _unique_url_index(db) == 1, "uq_git_sources_url must be back after the round-trip"