Skip to content

Config file

This page is generated from src/project.ts, so it says what the tool does.

An optional skill-evals.config.json in the project root sets defaults. A flag wins over the file.

Key Type Default of Meaning
cases string --cases <dir> The directory that holds the cases.
harness string --harness <ids|all> Comma-separated harness ids.
reps number --reps <n> Runs for each subject and case.
model string --model <id> Passed to the harness unchanged.
judgeHarness string --judge-harness <id> The harness that scores judged checks.
judgeModel string --judge-model <id> The model that scores judged checks.
decisionModel string --decision-model <id> The decision model that scores decision checks.
decisionUrl string --decision-url <url> The base URL of the decision model’s API.
decisionTimeout number --decision-timeout <seconds> How long one request to the decision model may take. A positive integer.
minScore number --min-score <0..1> The lowest suite score that passes. The command exits 1 when the suite score of the subject (B for run) is lower.

Every key is optional. cases resolves against the project root. harness narrows check too. An unknown key, a value of the wrong type, or invalid JSON is an error with exit code 2.

A file that sets every key:

{
"cases": "evals/cases",
"harness": "claude-code",
"reps": 3,
"model": "<model-id>",
"judgeHarness": "claude-code",
"judgeModel": "<model-id>",
"decisionModel": "<decision-model-id>",
"decisionUrl": "<base-url>",
"decisionTimeout": 60,
"minScore": 0.8
}