API
This page is generated from the doc comments of what src/index.ts exports, so it says what the tool does.
import { evaluate, loadCases, HARNESSES } from "skill-evals";Use it from code shows an evaluation from start to end. The package ships type declarations, so everything on this page is typed in TypeScript.
Functions and values
Section titled “Functions and values”loadCases()
Section titled “loadCases()”function loadCases(casesDir: string, glob?: string): Case[];Loads the cases of a suite.
| Parameter | Description |
|---|---|
casesDir |
A directory with one subdirectory for each case. |
glob |
Selects cases by id; only * is special. |
Returns. Every matching case, sorted by id.
Throws UsageError. When the directory cannot be read, a case is invalid, or no case matches.
evaluate()
Section titled “evaluate()”function evaluate(given: EvaluateOptions): Promise<Evaluation>;Runs every case against each subject, scores the runs, and writes results.json and each run’s files to the output directory. One subject is scored on its own. With two, the first is the baseline and the second is the one under test, which the pass bars apply to. It prints nothing and never calls process.exit. While it runs, SIGINT and SIGTERM end the process with exit code 130 after the cleanup.
| Parameter | Description |
|---|---|
given |
What to evaluate and how. An option that is not set takes its value from the project’s config file. |
Returns. The scores, every run record, and where the files are.
Throws UsageError. For a wrong option, case file, or config file, before any harness starts.
HARNESSES
Section titled “HARNESSES”The harness adapters in default-preference order: the first installed one runs when --harness is absent. Pi leads because it has the smallest install, the fastest start, and reports tokens and cost.
const HARNESSES: Harness[];One prompt and what a good run of it does.
| Property | Type | Description |
|---|---|---|
id |
string |
|
prompt |
string |
|
checks |
Check[] |
|
weight |
number |
|
minScore? |
number | undefined |
|
maxTurns? |
number | undefined |
|
timeoutSeconds |
number |
|
workspaceDir |
string | null |
A directory whose content the agent starts with, or null for an empty workspace. |
One thing a case expects of a run.
type Check = RuledCheck | JudgedCheck | DecisionCheck;Evaluation
Section titled “Evaluation”What evaluate returns. The evaluation succeeded when belowBar and invalid are both empty.
| Property | Type | Description |
|---|---|---|
cases |
Case[] |
|
outDir |
string |
|
subjects |
SubjectSummary[] |
|
harnesses |
string[] |
|
harnessReason |
string |
|
model |
string | null |
|
judge |
string |
The harness and model that scored the judged checks, in words. |
decisionModel |
string | null |
The model that scored the decision checks; null when none is set. |
reps |
number |
|
suites |
SuiteScore[] |
|
belowBar |
string[] |
The pass bars that were not reached. |
invalid |
string[] |
The runs that do not count, each with its reason. |
records |
RunRecord[] |
EvaluateOptions
Section titled “EvaluateOptions”What evaluate takes. An option that is not set takes its value from the project’s config file.
| Property | Type | Description |
|---|---|---|
cwd? |
string | undefined |
The working directory, which sets the project and what relative paths resolve against. Default: the process’s. |
subjects |
string[] |
One or two of a directory, a git ref of the project, or “none”. |
harness? |
string | undefined |
Comma-separated harness ids, or “all”. Default: the first installed harness. |
cases? |
string | undefined |
The directory that holds the cases. Default: evals/cases in the project. |
caseGlob? |
string | undefined |
Case ids to run; only * is special. Default: every case. |
reps? |
number | undefined |
Runs for each subject and case. Default: 3. |
model? |
string | undefined |
Passed to the harness. Default: the harness’s own default. |
judgeHarness? |
string | undefined |
The harness that scores judged checks. Default: the harness under test. |
judgeModel? |
string | undefined |
The model that scores judged checks. Default: model. |
decisionModel? |
string | undefined |
The model that scores decision checks. No default: a suite with a decision check needs it. |
decisionUrl? |
string | undefined |
The base URL of the decision model’s API. Default: the provider’s own. |
decisionTimeout? |
number | undefined |
The seconds one request to the decision model may take, a positive integer. Default: 60. |
minScore? |
number | undefined |
The pass bar for the suite score of the subject under test. |
outDir? |
string | undefined |
Where results go. Default: a new temp directory. |
onRun? |
((record: RunRecord) => void) | undefined |
Called with each run’s record as soon as the run ends. |
Results
Section titled “Results”What evaluate writes to results.json.
| Property | Type | Description |
|---|---|---|
subjects |
SubjectSummary[] |
|
harnesses |
string[] |
|
harnessReason |
string |
|
model |
string | null |
|
judge |
string |
The harness and model that scored the judged checks, in words. |
decisionModel |
string | null |
The model that scored the decision checks; null when none is set. |
reps |
number |
|
suites |
SuiteScore[] |
|
belowBar |
string[] |
The pass bars that were not reached. |
invalid |
string[] |
The runs that do not count, each with its reason. |
records |
RunRecord[] |
Harness
Section titled “Harness”The adapter of one coding-agent CLI.
| Property | Type | Description |
|---|---|---|
id |
string |
|
bin |
string |
The name of the executable on PATH. |
authFiles |
string[] |
Login files to copy into the throwaway home, relative to the user’s home. |
invocation |
(input: RunInput) => Invocation |
|
parse |
(stdout: string) => Parsed |
May throw on a stream it does not understand; the caller reports that as a failed run. |
Parsed
Section titled “Parsed”A harness’s output stream in the one shape every harness shares. Null means the harness does not report the field.
| Property | Type | Description |
|---|---|---|
finalText |
string |
|
toolCalls |
ToolCall[] |
|
tokens |
Tokens | null |
|
costUsd |
number | null |
|
loaded |
{ paths: string[] | null; skills: string[] | null; } |
What the harness says it loaded at startup: the directories of its plugins and the names of its skills. |
error |
string | null |
Tokens
Section titled “Tokens”Token counts; input counts every prompt token, cached ones included.
| Property | Type | Description |
|---|---|---|
input |
number |
|
output |
number |
ToolCall
Section titled “ToolCall”One tool call of the agent.
| Property | Type | Description |
|---|---|---|
name |
string |
The harness’s own tool name. |
kind |
"shell" | "read" | "write" | "edit" | "search" | "web" | "skill" | "subagent" | "other" |
|
input |
unknown |
CheckResult
Section titled “CheckResult”A check with what it gave one run.
type CheckResult = Check & { /** Null when a judged check or a decision check got no usable answer. */ score: number | null; /** The judge's reason, or why there is no score. */ reason?: string; /** For a decision check, the probability the model gave each allowed answer. */ probabilities?: Record<string, number>;};RunRecord
Section titled “RunRecord”Everything kept about one run.
| Property | Type | Description |
|---|---|---|
harness |
string |
|
subject |
string |
The subject’s label. |
caseId |
string |
|
rep |
number |
|
ok |
boolean |
|
error |
string | null |
|
finalText |
string |
|
toolCalls |
ToolCall[] |
|
skillsLoaded |
string[] |
|
subjectLoaded |
boolean | null |
Whether the harness reported the subject as loaded at startup; null when it reports nothing, or when there is nothing to load. |
checks |
CheckResult[] |
|
score |
number | null |
The run score, null when the run is invalid. |
judge |
(Usage & { calls: number; }) | null |
What the judged checks cost, kept apart from the run’s own numbers; null when no judge ran. |
decision |
(Usage & { calls: number; }) | null |
What the decision checks cost, kept apart in the same way; null when no request was sent. |
durationMs |
number |
|
tokens |
Tokens | null |
|
costUsd |
number | null |
CaseScore
Section titled “CaseScore”A case’s scores, keyed by subject label.
| Property | Type | Description |
|---|---|---|
id |
string |
|
weight |
number |
|
minScore? |
number | undefined |
|
scores |
Record<string, number | null> |
SuiteScore
Section titled “SuiteScore”One harness’s case scores, and its suite score keyed by subject label.
| Property | Type | Description |
|---|---|---|
harness |
string |
|
cases |
CaseScore[] |
|
suite |
Record<string, number | null> |
SubjectSummary
Section titled “SubjectSummary”The part of a subject that goes into results.json and the report.
| Property | Type | Description |
|---|---|---|
label |
Label |
|
spec |
string |
What the user named: “none”, a directory, or a git ref of the project. |
kind |
"plugin" | "skills" | "none" |
“plugin” is a directory with a harness’s plugin packaging, loaded the way that harness loads a local plugin; “skills” is a plain folder of skills, copied to where each harness discovers skills; “none” loads nothing. |
dir |
string | null |
Null for “none”. For a git ref it is a detached worktree. |
skills |
string[] |
The names of the skills found in dir, sorted. |
commit |
string | null |
Set when spec is a git ref. |