Skip to content

API

This page is generated from the doc comments of what src/index.ts exports, so it says what the tool does.

import { evaluate, loadCases, HARNESSES } from "skill-evals";

Use it from code shows an evaluation from start to end. The package ships type declarations, so everything on this page is typed in TypeScript.

function loadCases(casesDir: string, glob?: string): Case[];

Loads the cases of a suite.

Parameter Description
casesDir A directory with one subdirectory for each case.
glob Selects cases by id; only * is special.

Returns. Every matching case, sorted by id.

Throws UsageError. When the directory cannot be read, a case is invalid, or no case matches.

function evaluate(given: EvaluateOptions): Promise<Evaluation>;

Runs every case against each subject, scores the runs, and writes results.json and each run’s files to the output directory. One subject is scored on its own. With two, the first is the baseline and the second is the one under test, which the pass bars apply to. It prints nothing and never calls process.exit. While it runs, SIGINT and SIGTERM end the process with exit code 130 after the cleanup.

Parameter Description
given What to evaluate and how. An option that is not set takes its value from the project’s config file.

Returns. The scores, every run record, and where the files are.

Throws UsageError. For a wrong option, case file, or config file, before any harness starts.

The harness adapters in default-preference order: the first installed one runs when --harness is absent. Pi leads because it has the smallest install, the fastest start, and reports tokens and cost.

const HARNESSES: Harness[];

One prompt and what a good run of it does.

Property Type Description
id string
prompt string
checks Check[]
weight number
minScore? number | undefined
maxTurns? number | undefined
timeoutSeconds number
workspaceDir string | null A directory whose content the agent starts with, or null for an empty workspace.

One thing a case expects of a run.

type Check = RuledCheck | JudgedCheck | DecisionCheck;

What evaluate returns. The evaluation succeeded when belowBar and invalid are both empty.

Property Type Description
cases Case[]
outDir string
subjects SubjectSummary[]
harnesses string[]
harnessReason string
model string | null
judge string The harness and model that scored the judged checks, in words.
decisionModel string | null The model that scored the decision checks; null when none is set.
reps number
suites SuiteScore[]
belowBar string[] The pass bars that were not reached.
invalid string[] The runs that do not count, each with its reason.
records RunRecord[]

What evaluate takes. An option that is not set takes its value from the project’s config file.

Property Type Description
cwd? string | undefined The working directory, which sets the project and what relative paths resolve against. Default: the process’s.
subjects string[] One or two of a directory, a git ref of the project, or “none”.
harness? string | undefined Comma-separated harness ids, or “all”. Default: the first installed harness.
cases? string | undefined The directory that holds the cases. Default: evals/cases in the project.
caseGlob? string | undefined Case ids to run; only * is special. Default: every case.
reps? number | undefined Runs for each subject and case. Default: 3.
model? string | undefined Passed to the harness. Default: the harness’s own default.
judgeHarness? string | undefined The harness that scores judged checks. Default: the harness under test.
judgeModel? string | undefined The model that scores judged checks. Default: model.
decisionModel? string | undefined The model that scores decision checks. No default: a suite with a decision check needs it.
decisionUrl? string | undefined The base URL of the decision model’s API. Default: the provider’s own.
decisionTimeout? number | undefined The seconds one request to the decision model may take, a positive integer. Default: 60.
minScore? number | undefined The pass bar for the suite score of the subject under test.
outDir? string | undefined Where results go. Default: a new temp directory.
onRun? ((record: RunRecord) => void) | undefined Called with each run’s record as soon as the run ends.

What evaluate writes to results.json.

Property Type Description
subjects SubjectSummary[]
harnesses string[]
harnessReason string
model string | null
judge string The harness and model that scored the judged checks, in words.
decisionModel string | null The model that scored the decision checks; null when none is set.
reps number
suites SuiteScore[]
belowBar string[] The pass bars that were not reached.
invalid string[] The runs that do not count, each with its reason.
records RunRecord[]

The adapter of one coding-agent CLI.

Property Type Description
id string
bin string The name of the executable on PATH.
authFiles string[] Login files to copy into the throwaway home, relative to the user’s home.
invocation (input: RunInput) => Invocation
parse (stdout: string) => Parsed May throw on a stream it does not understand; the caller reports that as a failed run.

A harness’s output stream in the one shape every harness shares. Null means the harness does not report the field.

Property Type Description
finalText string
toolCalls ToolCall[]
tokens Tokens | null
costUsd number | null
loaded { paths: string[] | null; skills: string[] | null; } What the harness says it loaded at startup: the directories of its plugins and the names of its skills.
error string | null

Token counts; input counts every prompt token, cached ones included.

Property Type Description
input number
output number

One tool call of the agent.

Property Type Description
name string The harness’s own tool name.
kind "shell" | "read" | "write" | "edit" | "search" | "web" | "skill" | "subagent" | "other"
input unknown

A check with what it gave one run.

type CheckResult = Check & {
/** Null when a judged check or a decision check got no usable answer. */
score: number | null;
/** The judge's reason, or why there is no score. */
reason?: string;
/** For a decision check, the probability the model gave each allowed answer. */
probabilities?: Record<string, number>;
};

Everything kept about one run.

Property Type Description
harness string
subject string The subject’s label.
caseId string
rep number
ok boolean
error string | null
finalText string
toolCalls ToolCall[]
skillsLoaded string[]
subjectLoaded boolean | null Whether the harness reported the subject as loaded at startup; null when it reports nothing, or when there is nothing to load.
checks CheckResult[]
score number | null The run score, null when the run is invalid.
judge (Usage & { calls: number; }) | null What the judged checks cost, kept apart from the run’s own numbers; null when no judge ran.
decision (Usage & { calls: number; }) | null What the decision checks cost, kept apart in the same way; null when no request was sent.
durationMs number
tokens Tokens | null
costUsd number | null

A case’s scores, keyed by subject label.

Property Type Description
id string
weight number
minScore? number | undefined
scores Record<string, number | null>

One harness’s case scores, and its suite score keyed by subject label.

Property Type Description
harness string
cases CaseScore[]
suite Record<string, number | null>

The part of a subject that goes into results.json and the report.

Property Type Description
label Label
spec string What the user named: “none”, a directory, or a git ref of the project.
kind "plugin" | "skills" | "none" “plugin” is a directory with a harness’s plugin packaging, loaded the way that harness loads a local plugin; “skills” is a plain folder of skills, copied to where each harness discovers skills; “none” loads nothing.
dir string | null Null for “none”. For a git ref it is a detached worktree.
skills string[] The names of the skills found in dir, sorted.
commit string | null Set when spec is a git ref.