Commands and flags
This page is generated from src/cli.ts and src/usage.ts, so it says what the tool does.
A wrong command line prints the same commands and flags as its help text.
Commands
Section titled “Commands”| Command | What it does |
|---|---|
run |
Run every case against subject A and subject B, score both, and print a comparison. |
score |
Run every case against one subject and print its scores. |
check |
Run one small prompt in each installed harness and show what its adapter reports. |
list |
Show the harnesses in default-preference order, and the cases. |
| Flag | Commands | Default | Meaning |
|---|---|---|---|
--a <subject> |
run |
main | The baseline. |
--b <subject> |
run |
the project, uncommitted changes included | The change. |
--subject <subject> |
score, check |
the project, uncommitted changes included | The subject to score or to probe. |
--harness <ids|all> |
run, score, check |
the first installed one; all for check | Comma-separated harness ids. |
--case <glob> |
run, score |
* | Which case ids to run. Only * is special. |
--cases <dir> |
run, score, list |
evals/cases in the project | The directory that holds the cases. |
--reps <n> |
run, score |
3 | Runs for each subject and case. |
--model <id> |
run, score, check |
the harness’s own default | Passed to the harness unchanged. |
--min-score <0..1> |
run, score |
no bar | The lowest suite score that passes. The command exits 1 when the suite score of the subject (B for run) is lower. |
--judge-harness <id> |
run, score |
the harness under test | The harness that scores judged checks. |
--judge-model <id> |
run, score |
the value of –model | The model that scores judged checks. |
--decision-model <id> |
run, score |
none | The decision model that scores decision checks. |
--decision-url <url> |
run, score |
the provider’s own | The base URL of the decision model’s API. |
--decision-timeout <seconds> |
run, score |
60 | How long one request to the decision model may take. A positive integer. |
--out <dir> |
run, score, check |
a new temp directory | Where results go. |
A subject is what a run loads: a directory, a git ref of the project, or none. Subjects has the details. Set a pass bar explains --min-score and the exit codes, and Decision checks explains the decision model’s flags.