Use it in CI
run and score can gate a pull request: the exit code says whether the suite passed.
| Exit code | Meaning |
|---|---|
| 0 | Every run is valid and every pass bar is reached. |
| 1 | A run failed or is invalid, or a score is under its bar. |
| 2 | A wrong command line, case file, config file, or subject. A decision check with no decision model or no API key. Also Windows, which is not supported. |
- Set the bar.
--min-score 0.8makes the command exit 1 when the suite score is under 0.8, andminScorein acase.jsondoes the same for one case. See Set a pass bar. - Name the model. Each run starts in a new, empty home directory, so no settings and no default model are there. Pass
--model, or setmodelin the config file. Without it each harness uses its built-in default, which may be a provider you have no key for. - Give the harness a login. A run copies the harness’s credential files into its new home. A login stored in the operating system’s keychain may not follow. Run
skill-evals checkon the machine first, and if it reports a login error, set the harness’s API key variable. - Keep the results.
--out <dir>sets where results go. The default is a new temp directory. Read the report says what the directory holds. - Review the prompts. Every run is unattended and has full permissions in its harness, so only run prompts you wrote or read. See Safety.