evals for agent skills

Know what your skills actually do.

An agent does not do the same thing twice. skill-evals runs your skills across the scenarios you care about, in a real coding agent, and scores each run against what you defined as success. Invoke a skill and see what it does, or leave it out of the prompt and see whether the agent reaches for it. Change a skill, run it again, and see what the change did.

npx skill-evals run --a none --b ./skills
Get started

Runs are unattended and have full permissions in the agent. Only run prompts you wrote or read.

skill comparison · claude-code · 3 runs each
Scores of three cases without and with a commit-message skill
case without skill with skill change what moved
staged-diff 0.33 1.00 +0.67 typed subject line 0/3 → 3/3
described-change 0.26 0.11 -0.15 skill loaded in 0 of 3 runs
git-question 1.00 1.00 0.00 skill stayed out, as it should
suite score 0.37 to 0.60 pass bar 1.00 on git-question · passed

A real report from a commit-message skill, condensed. It asks whether the skill loads: no prompt names it. The second row is the finding: the skill works, and it does not trigger when a change is only described.

Reading a skill is not running it

understand

What it does, not what it says

A language model can load a skill on one run and ignore it on the next. Run each scenario several times and see what happens, and how often.

define

Say what success is

Write down what a good result looks like for each scenario. That turns "it seems better" into a benchmark you can score against.

change

See what an edit did

Reword a description, add a rule, swap the model. Run the same scenarios again and compare the scores, before you merge.

Two questions you can ask

behavior

What does it do?

Write a prompt that invokes the skill, as you would in real use. You learn whether the work comes out the way you defined success.

triggering

Does it load?

Write a prompt that does not name the skill. You learn whether the agent picks it up on its own, and leaves it alone when it should.

How it works

01 · write a case

A scenario and its checks

A case is a folder: the prompt a user would type, and a JSON list of checks that say what a good result is. No test code.

02 · run it, more than once

A real agent, a clean home

Each case runs several times, with and without the skill. Every run starts your coding agent in a throwaway home directory, so your own settings and installed skills stay out.

03 · read the score

0 to 1, with a pass bar

Checks roll up to a score for each case and the suite, and every answer is saved so you can see why. Set a minimum and the command fails under it.

Three kinds of check

Deterministic

Did the skill load? Was a tool called? Does the file exist, and does it match? Scores 0 or 1, the same every time.

{ "type": "skill_loaded",
  "skill": "commit-message" }

Decision model

You define the answers and what each is worth. The model returns a probability for each, and the score is computed from them.

{ "type": "decision",
  "question": "Avoids jargon?",
  "expect": "yes" }

Judged

A model grades the answer against a rubric you write in plain text, for what a pattern cannot see.

{ "type": "judge",
  "rubric": "Keeps the meaning." }

Runs in the agent you use

One case format for all five. Score a plugin, a plain folder of skills, or nothing at all as the baseline.

Change a skill. Know what changed.

Open source, MIT, no dependencies. Node 22 or later and one installed agent, on macOS or Linux.