brain-of-reese/.agents/skills/test-embed-model/SKILL.md

name, description

name

description

test-embed-model

Tests an embedding model (BOR_LLM_EMBED_MODEL in .env) across three dimensions — dimension consistency, cosine accuracy on semantic pairs, and embedding speed — then records results in benchmarks/model_benchmarks.csv. Use when the user asks to test or benchmark an embedding model, check embedding quality, or compare embedding models (e.g. "test the embed model", "check embedding dimension", "benchmark embed speed").

Test an Embedding Model (dimension + cosine + speed)

Tests the configured BOR_LLM_EMBED_MODEL (default embed) against three independent checks:

1. Dimension check — output vector length matches BOR_EMBEDDING_DIM
2. Cosine accuracy — semantically similar text pairs have higher cosine similarity than dissimilar pairs (5 pairs tested)
3. Speed — vectors produced per second (50 vectors)

Rules (non-negotiable)

• Do not touch the semantic pair texts — they are the controlled benchmark corpus. Changing them is a methodology change: flag it.
• Do not edit app code (app/, tests/). This skill tests a model, not the app. If the model exposes an app defect, report it — don't fix it.
• .env is gitignored and is not committed — the model switch stays a live dev setting, and the summary must say which model .env is left on (default: the tested model).

Procedure

All commands run from the repo root with uv run.

1. Preconditions

podman compose up -d db
grep -E "BOR_LLM_(BASE_URL|API_KEY|EMBED_MODEL|EMBEDDING_DIM)" .env

2. Switch the model (optional)

Edit only BOR_LLM_EMBED_MODEL in .env (leave chat and summary alone):

BOR_LLM_EMBED_MODEL=<model>

3. Run the benchmark

# Single run (all three checks)
uv run python -m scripts.test_embed_model

# Specify a model explicitly
uv run python -m scripts.test_embed_model --model embed-v2

# Multiple runs for variance
uv run python -m scripts.test_embed_model --runs 3

Wall time: ~0.5–2 s per run (fast — the embedding endpoint is lightweight).

4. Read the verdicts

Each check prints ✓ (PASS) or ✗ (FAIL):

• dimension: actual_dim == expected_dim (from BOR_EMBEDDING_DIM, default 768)
• cosine: similar-pair margin ≥ 0.05 (similar > dissimilar by at least 0.05 cosine)
• speed: no NaN/Inf vectors in output; rate reported as vec/s

Gate: PASS if all three checks pass.

5. Record + commit

Results are automatically appended to benchmarks/model_benchmarks.csv (one row per check per run). Commit:

git add benchmarks/model_benchmarks.csv
git commit --no-gpg-sign \
  -m "docs(agent): record the <model> embedding benchmark" \
  -m "<one-paragraph body: dimension match, cosine margin, speed, any failures>"

6. Summary

Table of dimension / cosine margin / speed per run, the one-line conclusion, and a note that .env is now on <model>.

Troubleshooting

• dimension mismatch — the model's output vectors are a different size than expected. This is a configuration error: check the model's docs for its embedding dimension and update BOR_EMBEDDING_DIM.
• cosine margin < 0.05 — the embeddings don't separate similar from dissimilar texts well. The model may be a poor embedding model or the semantic pairs are too generic. Report the margin.
• NaN/Inf vectors — the model's embedding endpoint is broken or the input text is malformed. Check the endpoint directly.
• very slow — >100 vec/s is typical for a local endpoint; remote endpoints may be slower due to network latency. Report the rate.