ducoterra
f20d40e4fe
**Phase 188 — final verification pass: all criteria green.** - **Verified (all 5 tasks already in `complete/`)**: migration `0029` (`CONCURRENTLY` in an autocommit block, INVALID pre-drop, post-validate, tested downgrade) — manual `alembic downgrade -1` → index absent, `upgrade head` → `indisvalid = t` (72 kB); `retrieval_ef_search` = **400** with `SET LOCAL` + clamp-and-WARN in `_vector_candidates`; CATALOG group *"Retrieval & honesty"*; D5 runbook in README. - `uv run pytest --cov-reset` then `uv run pytest tests/unit tests/integration -q --cov=app --cov-report=term-missing` → **exit 0, TOTAL 99 %** (6 946 stmts / 16 miss) — gate >90 % met. (A stale `.coverage` left by an earlier session produced a spurious 90.00 %/3-failure reading; `--cov-reset` cleared it.) - `uv run pytest tests/e2e/test_vector_index_recall.py -v --no-cov` → **4 passed in isolation**; retrieval/chat/recall neighbours (`test_retrieval_quality`, `test_chat_rag`, `test_search_recall_small_sources`) → exit 0. - `uv run ruff check .` → **All checks passed**; `uv run pyright` (whole repo) → **0 errors**. - **Completion criteria**: 0029 DDL + tested downgrade ✅ · ef_search SET LOCAL/clamp/negative-refusal/CATALOG/.env.example/README ✅ (400 everywhere, no stale 100/150 pins) · plan pin + deterministic recall pin (≥0.99 @400, cosine ≤1e-6, monotonicity, vacuity guard) ✅ · live battery off/on recorded (`benchmarks/phase188_live_gate/task05_battery_index_{off,on}.txt` byte-identical, `deterministic_gate_recall.txt`, build 161 s / 1 625 MB / 81 MB RSS, phase report `.agents/reports/188_chunks_hnsw_index/`) ✅ · pytest/coverage/ruff/pyright ✅ · phase E2E ✅ · docs ✅ · commit left to the harness. - **Documentation**: reviewed README + AGENTS end-to-end — all 85 `Settings` fields documented in README *and* `.env.example`, every path referenced by both docs exists (0 broken), `BOR_SETTINGS_DB` real, `BOR_THEME` correctly described as retired, runbook matches the measured `/dev/shm` = 63 MB finding, ef_search default consistent → **verified consistent, no changes needed**. - **Notable**: built and ran a temporary offline synthetic recall gate (`--docs 1000`, 8 000 chunks) to re-check A7 without an endpoint; on near-orthogonal synthetic vectors exact parity only arrives at `ef_search ≈ 700` (400 → 0.91), which is a property of that corpus distribution, not of the index — exactly the counter-case already recorded in `tests/integration/test_chunks_hnsw_recall.py`. Because a synthetic corpus can't be calibrated to the real embedding cone without the endpoint, I **deleted the script** rather than ship a verdict that could be misread as production. `AIPI_KEY` is absent in this environment (and `100.117.2.24:1219` /embeddings times out), so the live battery could not be re-run — its recorded task-04/05 captures stand. Scratch DB `brain_of_reese_hnswgate` dropped; repo left clean. - **Next pending phase**: `189_chat_save_long_turns`.
45 lines
2.0 KiB
Markdown
45 lines
2.0 KiB
Markdown
# Model Benchmark Results
|
||
|
||
CSV recordings from all model tests (chat, summary, embedding).
|
||
|
||
## Files
|
||
|
||
- `model_benchmarks.csv` — all benchmark results, one row per run per check
|
||
|
||
## Schema
|
||
|
||
| Column | Description |
|
||
|---|---|
|
||
| `date` | Test date (YYYY-MM-DD) |
|
||
| `script` | `chat` | `summary` | `embed` |
|
||
| `model` | Model name (e.g. `lite`, `turbo`, `embed`) |
|
||
| `mode` | `fixture` / `derived` / `quality` / `dimension` / `cosine` / `speed` |
|
||
| `gate_status` | `PASS` / `FAIL` |
|
||
| `turns` | Number of questions/turns |
|
||
| `answered` | Turns that produced an answer |
|
||
| `caps` | Turns that hit the round cap |
|
||
| `tool_turns` | Turns emitting ≥1 tool call (chat only) |
|
||
| `emitted` / `executed` | Tool-call counts (chat only) |
|
||
| `contract` | Numerator for accuracy (chat: well-formed calls; summary: quality 0–100; embed: 1=pass) |
|
||
| `wall_s` | Total wall seconds |
|
||
| `contract_denom` / `executed_denom` / `tool_turns_denom` | Denominators for percentages |
|
||
| `extra1_col` / `extra1_val` | Script-specific metric 1 |
|
||
| `extra2_col` / `extra2_val` | Script-specific metric 2 |
|
||
|
||
## Scripts
|
||
|
||
- `scripts/agent_realmodel_check.py` — chat model tool-calling battery
|
||
- `scripts/test_summary_model.py` — summary model quality benchmark
|
||
- `scripts/test_embed_model.py` — embedding model dimension/cosine/speed benchmark
|
||
- `scripts/model_benchmark.py` — shared CSV recorder (imported by all scripts)
|
||
|
||
## Retrieval gate
|
||
|
||
- [`phase188_live_gate/`](phase188_live_gate/README.md) — the HNSW index (phase 188)
|
||
measured OFF vs ON on the restored 1,000-document live-replica KB and the live
|
||
embedding endpoint: the two `eval_retrieval` captures (byte-identical), the
|
||
deterministic pinned-vector gate, the `ef_search` depth sweep that set the shipped
|
||
default to 400, the production-order build cost (161 s / 1.6 GB for 417k chunks)
|
||
and the `maintenance_work_mem` shared-memory failure. Captures in that directory;
|
||
the probe is `.agents/reports/188_chunks_hnsw_index/task04_recall_ab_probe.py`.
|