understand
What it does, not what it says
A language model can load a skill on one run and ignore it on the next. Run each scenario several times and see what happens, and how often.
evals for agent skills
An agent does not do the same thing twice. skill-evals runs your skills across the scenarios you care about, in a real coding agent, and scores each run against what you defined as success. Invoke a skill and see what it does, or leave it out of the prompt and see whether the agent reaches for it. Change a skill, run it again, and see what the change did.
npx skill-evals run --a none
--b ./skills
Runs are unattended and have full permissions in the agent. Only run prompts you wrote or read.
| case | without skill | with skill | change | what moved |
|---|---|---|---|---|
| staged-diff | 0.33 | 1.00 | +0.67 | typed subject line 0/3 → 3/3 |
| described-change | 0.26 | 0.11 | -0.15 | skill loaded in 0 of 3 runs |
| git-question | 1.00 | 1.00 | 0.00 | skill stayed out, as it should |
A real report from a commit-message skill, condensed. It asks whether the skill loads: no prompt names it. The second row is the finding: the skill works, and it does not trigger when a change is only described.
understand
A language model can load a skill on one run and ignore it on the next. Run each scenario several times and see what happens, and how often.
define
Write down what a good result looks like for each scenario. That turns "it seems better" into a benchmark you can score against.
change
Reword a description, add a rule, swap the model. Run the same scenarios again and compare the scores, before you merge.
behavior
Write a prompt that invokes the skill, as you would in real use. You learn whether the work comes out the way you defined success.
triggering
Write a prompt that does not name the skill. You learn whether the agent picks it up on its own, and leaves it alone when it should.
01 · write a case
A case is a folder: the prompt a user would type, and a JSON list of checks that say what a good result is. No test code.
02 · run it, more than once
Each case runs several times, with and without the skill. Every run starts your coding agent in a throwaway home directory, so your own settings and installed skills stay out.
03 · read the score
Checks roll up to a score for each case and the suite, and every answer is saved so you can see why. Set a minimum and the command fails under it.
Did the skill load? Was a tool called? Does the file exist, and does it match? Scores 0 or 1, the same every time.
{ "type": "skill_loaded",
"skill": "commit-message" }
You define the answers and what each is worth. The model returns a probability for each, and the score is computed from them.
{ "type": "decision",
"question": "Avoids jargon?",
"expect": "yes" }
A model grades the answer against a rubric you write in plain text, for what a pattern cannot see.
{ "type": "judge",
"rubric": "Keeps the meaning." }
One case format for all five. Score a plugin, a plain folder of skills, or nothing at all as the baseline.
Open source, MIT, no dependencies. Node 22 or later and one installed agent, on macOS or Linux.