# Model Benchmark Results

CSV recordings from all model tests (chat, summary, embedding).

## Files

- `model_benchmarks.csv` — all benchmark results, one row per run per check

## Schema

| Column | Description |
|---|---|
| `date` | Test date (YYYY-MM-DD) |
| `script` | `chat` | `summary` | `embed` |
| `model` | Model name (e.g. `lite`, `turbo`, `embed`) |
| `mode` | `fixture` / `derived` / `quality` / `dimension` / `cosine` / `speed` |
| `gate_status` | `PASS` / `FAIL` |
| `turns` | Number of questions/turns |
| `answered` | Turns that produced an answer |
| `caps` | Turns that hit the round cap |
| `tool_turns` | Turns emitting ≥1 tool call (chat only) |
| `emitted` / `executed` | Tool-call counts (chat only) |
| `contract` | Numerator for accuracy (chat: well-formed calls; summary: quality 0–100; embed: 1=pass) |
| `wall_s` | Total wall seconds |
| `contract_denom` / `executed_denom` / `tool_turns_denom` | Denominators for percentages |
| `extra1_col` / `extra1_val` | Script-specific metric 1 |
| `extra2_col` / `extra2_val` | Script-specific metric 2 |

## Scripts

- `scripts/agent_realmodel_check.py` — chat model tool-calling battery
- `scripts/test_summary_model.py` — summary model quality benchmark
- `scripts/test_embed_model.py` — embedding model dimension/cosine/speed benchmark
- `scripts/model_benchmark.py` — shared CSV recorder (imported by all scripts)

## Retrieval gate

- [`phase188_live_gate/`](phase188_live_gate/README.md) — the HNSW index (phase 188)
  measured OFF vs ON on the restored 1,000-document live-replica KB and the live
  embedding endpoint: the two `eval_retrieval` captures (byte-identical), the
  deterministic pinned-vector gate, the `ef_search` depth sweep that set the shipped
  default to 400, the production-order build cost (161 s / 1.6 GB for 417k chunks)
  and the `maintenance_work_mem` shared-memory failure. Captures in that directory;
  the probe is `.agents/reports/188_chunks_hnsw_index/task04_recall_ab_probe.py`.