Overview
Know what your skills actually do. skill-evals runs your skills across the scenarios you care about, in a real coding agent, and scores each run from 0 to 1 against what you defined as success.
A language model does not do the same thing twice, so reading a skill tells you what you wrote and not what the agent will do with it. An eval is how you find out:
- Understand. Run each scenario several times and see what the agent does, and how often. It may never load the skill, load it and do the same as before, or load it for requests that have nothing to do with it.
- Define success. Write down what a good result is for each scenario. That turns “it seems better” into a benchmark you can score against.
- Change with evidence. Edit the skill, run the same scenarios again, and compare the scores before you merge.
There are two questions to ask of a skill, and each case asks one of them:
- What does it do when it runs? The prompt invokes the skill by name, the way you would in real use, and the checks are about the result.
- Does it load when it should? The prompt does not name the skill, and the checks are whether the agent picks it up on its own, and stays out when it should.
This is a report for one commit-message skill, in Claude Code, with three runs for each side. A is no skills and B is the folder that holds the skill. It asks the second question: no prompt names the skill.
| Case | Score, A / B (B - A) | What moved |
|---|---|---|
staged-diff |
0.33 / 1.00 (+0.67) | The skill loaded in 0 of 3 runs on A and 3 of 3 on B. A typed subject line: 0 of 3, then 3 of 3. |
described-change |
0.26 / 0.11 (-0.15) | The skill loaded in 0 of 3 runs on B. |
git-question |
1.00 / 1.00 (+0.00) | A question that is not about writing a commit message. The skill stayed out in 3 of 3 runs on both. |
Suite score: 0.37 / 0.60 (+0.23)
The numbers are from a real run. The table is shortened: the report that the command prints shows every check of a case with its own pass count, and the duration, tokens, and cost of each case. The second row is the finding. The skill works when the agent loads it, and the agent does not load it when a change is only described.
How it works
Section titled “How it works”- You write a case. A case is a folder under
evals/cases/with aprompt.md, which holds the prompt exactly as a user would type it, and acase.json, which holds the checks. An optionalworkspace/folder holds files the agent finds in its working directory. - Each case runs on both sides. A run starts the harness in a new, empty home directory and a new workspace, with only the subject loaded: your skills, a plugin, or nothing. Your login is copied in. Your settings and the skills you installed globally are not. Each side runs three times unless you set
--reps. - Checks become scores. Every check scores a run from 0 to 1. The checks of a run give a run score, the runs give a case score, and the cases give a suite score. Weights and pass bars are yours to set.
A run that failed or is invalid does not count. It is listed under the table and makes the command exit 1. What one run does has every step.
Three kinds of check
Section titled “Three kinds of check”Deterministic. Did the skill load? Was a tool called? Does a file exist, and does it match a pattern? The score is 0 or 1.
{ "type": "skill_loaded", "skill": "commit-message" }Decision model. You set the allowed answers and what each is worth. A decision model returns a probability for each answer, and the score is computed from the probabilities. No model writes the score.
{ "type": "decision", "question": "Does the answer avoid engineering jargon?", "expect": "yes"}Judged. A model scores the final answer against a rubric that you write in plain text. Use it for a quality that no pattern can find. A judge’s score moves between runs, so use at least three reps.
{ "type": "judge", "rubric": "The restatement keeps the meaning of the original. It has at most three sentences."}Checks lists every check type and its fields.
Harnesses
Section titled “Harnesses”One case format works in five coding agents: Pi, Claude Code, Cursor, OpenCode, and Codex. Harnesses lists their ids in the order run and score prefer them.
The harnesses are not equally proven. Only Claude Code ran a real model while the adapters were written, and the Cursor adapter has not run past the login check. What was verified says what was confirmed for each one, and how.
Quick start gets a first score for both questions. Read Safety before you run it.