brain-of-reese/.agents/skills/test-summary-model/SKILL.md

name, description

name

description

test-summary-model

Tests a summary model (BOR_LLM_SUMMARY_MODEL in .env) against a fixed set of 8 source texts from the fixture KB — evaluates coherence, coverage, brevity, and hallucination detection — then records results in benchmarks/model_benchmarks.csv. Use when the user asks to test or benchmark a summary model, compare summary models, or evaluate summary quality (e.g. "test lite summary", "compare summary models", "how good is turbo at summarizing").

Test a Summary Model (quality benchmark)

Tests the configured BOR_LLM_SUMMARY_MODEL (default lite) against 8 source texts extracted from the fixture KB. Each text is a realistic documentation excerpt (~300–500 chars). The model is asked to summarize each in 2–4 sentences.

Rules (non-negotiable)

• Do not touch the fixture source texts — they are the controlled benchmark corpus. Changing them is a methodology change: flag it.
• Do not edit app code (app/, tests/). This skill tests a model, not the app. If the model exposes an app defect, report it — don't fix it.
• .env is gitignored and is not committed — the model switch stays a live dev setting, and the summary must say which model .env is left on (default: the tested model).

Procedure

All commands run from the repo root with uv run.

1. Preconditions

podman compose up -d db
grep -E "BOR_LLM_(BASE_URL|API_KEY|SUMMARY_MODEL)" .env

2. Switch the model (optional)

Edit only BOR_LLM_SUMMARY_MODEL in .env (leave chat and embed alone):

BOR_LLM_SUMMARY_MODEL=<model>

3. Run the benchmark

# Single run
uv run python -m scripts.test_summary_model

# Specify a model explicitly
uv run python -m scripts.test_summary_model --model turbo

# Multiple runs for variance
uv run python -m scripts.test_summary_model --runs 3

Wall time: ~3–5 s per text, ~25–40 s total for 8 texts.

4. Read the verdicts

The gate: line: PASS|FAIL turns=N answered=N quality=Q hallucinations=H/N coherence=C/5 coverage=V/5 brevity=B/5 (wall Ts).

Scoring rubric:

• coherence (0–5): non-empty, multi-sentence, starts with capital
• coverage (0–5): captures ≥2 key numbers/facts from source
• brevity (0–5): summary length / source length ratio 0.10–0.25 = 5, 0.05–0.35 = 4, etc.
• hallucination: detected when summary contains >3 uncommon tokens not in source

Quality score = coherence×0.35 + coverage×0.40 + brevity×0.25, scaled 0–100, with a 5-point penalty per hallucination.

Gate: PASS if quality ≥ 70 and hallucination rate < 25%.

5. Record + commit

Results are automatically appended to benchmarks/model_benchmarks.csv. Also append a summary line to benchmarks/README.md if it exists, or create it:

git add benchmarks/model_benchmarks.csv
git commit --no-gpg-sign \
  -m "docs(agent): record the <model> summary benchmark" \
  -m "<one-paragraph body: quality score, hallucination count, wall time, comparison to other models>"

6. Summary

Table of quality / hallucinations / coherence / coverage / brevity / wall per run, the one-line conclusion, and a note that .env is now on <model>.

Troubleshooting

• endpoint down / all turns error — check curl -s $BOR_LLM_BASE_URL/models with the key; the gate will exit 1 with answered<8. Report, don't retry-loop.
• low brevity — the model is restating rather than condensing. This is model-specific; try a more explicit prompt or a different model.
• hallucinations — the model is adding details not in the source. This is the most common failure mode; tighten the prompt or switch models.