ducoterra
f221b40fce
New files: - scripts/model_benchmark.py — shared CSV recorder for all model tests - scripts/test_summary_model.py — summary model quality benchmark (coherence, coverage, brevity, hallucination) - scripts/test_embed_model.py — embedding model benchmark (dimension, cosine accuracy, speed) - .agents/skills/test-summary-model/SKILL.md — skill for testing summary models - .agents/skills/test-embed-model/SKILL.md — skill for testing embedding models - benchmarks/README.md — schema documentation Updated: - .agents/skills/test-chat-model/SKILL.md — now also records to CSV All three scripts write to benchmarks/model_benchmarks.csv with one row per run per check. The CSV accumulates results across runs for comparison.
99 lines
3.6 KiB
Markdown
99 lines
3.6 KiB
Markdown
---
|
||
name: test-embed-model
|
||
description: Tests an embedding model (BOR_LLM_EMBED_MODEL in .env) across three dimensions — dimension consistency, cosine accuracy on semantic pairs, and embedding speed — then records results in benchmarks/model_benchmarks.csv. Use when the user asks to test or benchmark an embedding model, check embedding quality, or compare embedding models (e.g. "test the embed model", "check embedding dimension", "benchmark embed speed").
|
||
---
|
||
|
||
# Test an Embedding Model (dimension + cosine + speed)
|
||
|
||
Tests the configured `BOR_LLM_EMBED_MODEL` (default `embed`) against
|
||
three independent checks:
|
||
|
||
1. **Dimension check** — output vector length matches `BOR_EMBEDDING_DIM`
|
||
2. **Cosine accuracy** — semantically similar text pairs have higher
|
||
cosine similarity than dissimilar pairs (5 pairs tested)
|
||
3. **Speed** — vectors produced per second (50 vectors)
|
||
|
||
## Rules (non-negotiable)
|
||
|
||
- **Do not touch** the semantic pair texts — they are the controlled
|
||
benchmark corpus. Changing them is a methodology change: flag it.
|
||
- **Do not edit app code** (`app/`, `tests/`). This skill tests a model,
|
||
not the app. If the model exposes an app defect, report it — don't fix it.
|
||
- `.env` is gitignored and is **not committed** — the model switch stays
|
||
a live dev setting, and the summary must say which model `.env` is left
|
||
on (default: the tested model).
|
||
|
||
## Procedure
|
||
|
||
All commands run from the repo root with `uv run`.
|
||
|
||
### 1. Preconditions
|
||
|
||
```bash
|
||
podman compose up -d db
|
||
grep -E "BOR_LLM_(BASE_URL|API_KEY|EMBED_MODEL|EMBEDDING_DIM)" .env
|
||
```
|
||
|
||
### 2. Switch the model (optional)
|
||
|
||
Edit only `BOR_LLM_EMBED_MODEL` in `.env` (leave chat and summary alone):
|
||
|
||
```
|
||
BOR_LLM_EMBED_MODEL=<model>
|
||
```
|
||
|
||
### 3. Run the benchmark
|
||
|
||
```bash
|
||
# Single run (all three checks)
|
||
uv run python -m scripts.test_embed_model
|
||
|
||
# Specify a model explicitly
|
||
uv run python -m scripts.test_embed_model --model embed-v2
|
||
|
||
# Multiple runs for variance
|
||
uv run python -m scripts.test_embed_model --runs 3
|
||
```
|
||
|
||
Wall time: ~0.5–2 s per run (fast — the embedding endpoint is lightweight).
|
||
|
||
### 4. Read the verdicts
|
||
|
||
Each check prints `✓` (PASS) or `✗` (FAIL):
|
||
|
||
- **dimension**: `actual_dim == expected_dim` (from `BOR_EMBEDDING_DIM`, default 768)
|
||
- **cosine**: similar-pair margin ≥ 0.05 (similar > dissimilar by at least 0.05 cosine)
|
||
- **speed**: no NaN/Inf vectors in output; rate reported as vec/s
|
||
|
||
Gate: PASS if all three checks pass.
|
||
|
||
### 5. Record + commit
|
||
|
||
Results are automatically appended to `benchmarks/model_benchmarks.csv`
|
||
(one row per check per run). Commit:
|
||
|
||
```bash
|
||
git add benchmarks/model_benchmarks.csv
|
||
git commit --no-gpg-sign \
|
||
-m "docs(agent): record the <model> embedding benchmark" \
|
||
-m "<one-paragraph body: dimension match, cosine margin, speed, any failures>"
|
||
```
|
||
|
||
### 6. Summary
|
||
|
||
Table of dimension / cosine margin / speed per run, the one-line
|
||
conclusion, and a note that `.env` is now on `<model>`.
|
||
|
||
## Troubleshooting
|
||
|
||
- **dimension mismatch** — the model's output vectors are a different
|
||
size than expected. This is a configuration error: check the model's
|
||
docs for its embedding dimension and update `BOR_EMBEDDING_DIM`.
|
||
- **cosine margin < 0.05** — the embeddings don't separate similar from
|
||
dissimilar texts well. The model may be a poor embedding model or
|
||
the semantic pairs are too generic. Report the margin.
|
||
- **NaN/Inf vectors** — the model's embedding endpoint is broken or
|
||
the input text is malformed. Check the endpoint directly.
|
||
- **very slow** — >100 vec/s is typical for a local endpoint; remote
|
||
endpoints may be slower due to network latency. Report the rate.
|