Skip to content

Commands and flags

This page is generated from src/cli.ts and src/usage.ts, so it says what the tool does.

A wrong command line prints the same commands and flags as its help text.

Command What it does
run Run every case against subject A and subject B, score both, and print a comparison.
score Run every case against one subject and print its scores.
check Run one small prompt in each installed harness and show what its adapter reports.
list Show the harnesses in default-preference order, and the cases.
Flag Commands Default Meaning
--a <subject> run main The baseline.
--b <subject> run the project, uncommitted changes included The change.
--subject <subject> score, check the project, uncommitted changes included The subject to score or to probe.
--harness <ids|all> run, score, check the first installed one; all for check Comma-separated harness ids.
--case <glob> run, score * Which case ids to run. Only * is special.
--cases <dir> run, score, list evals/cases in the project The directory that holds the cases.
--reps <n> run, score 3 Runs for each subject and case.
--model <id> run, score, check the harness’s own default Passed to the harness unchanged.
--min-score <0..1> run, score no bar The lowest suite score that passes. The command exits 1 when the suite score of the subject (B for run) is lower.
--judge-harness <id> run, score the harness under test The harness that scores judged checks.
--judge-model <id> run, score the value of –model The model that scores judged checks.
--decision-model <id> run, score none The decision model that scores decision checks.
--decision-url <url> run, score the provider’s own The base URL of the decision model’s API.
--decision-timeout <seconds> run, score 60 How long one request to the decision model may take. A positive integer.
--out <dir> run, score, check a new temp directory Where results go.

A subject is what a run loads: a directory, a git ref of the project, or none. Subjects has the details. Set a pass bar explains --min-score and the exit codes, and Decision checks explains the decision model’s flags.