Checks
This page is generated from src/checks.ts, so it says what the tool does.
A check is one object in the checks list of a case.json. Every check has a type, and takes an optional weight, a positive number that defaults to 1. Write a case shows how to use them.
Check types
Section titled “Check types”| Check | Fields | Scores 1 when |
|---|---|---|
skill_loaded |
skill |
The agent loaded that skill. |
skill_not_loaded |
skill |
The agent did not load that skill. |
output_matches |
pattern, optional flags |
The final answer matches the regular expression. |
output_lacks |
pattern, optional flags |
The final answer does not match the regular expression. |
output_contains |
text |
The final answer contains that text. |
output_not_contains |
text |
The final answer does not contain that text. |
file_exists |
path |
That file exists in the workspace after the run. |
file_matches |
path, pattern, optional flags |
That file exists and matches the regular expression. |
tool_called |
at least one of kind, name, optional inputMatches, optional flags |
The agent made a tool call that fits every given field. |
tool_not_called |
at least one of kind, name, optional inputMatches, optional flags |
The agent made no tool call that fits every given field. |
judge |
rubric |
Not 0 or 1. A model scores the final answer against the rubric. |
decision |
question, optional expect. It takes exactly one of expect and options. |
Not 0 or 1. A decision model gives each allowed answer a probability, and the score is the expected value. |
Fields
Section titled “Fields”| Field | Holds |
|---|---|
skill |
The name of a skill. |
pattern |
A regular expression. |
text |
Text to look for. The match is exact and case-sensitive. |
path |
A file path inside the workspace. |
kind |
What the tool does, with the same word in every harness. |
name |
A regular expression for the harness’s own tool name. |
inputMatches |
A regular expression for the tool call’s input, written as JSON. |
flags |
The flags of the check’s regular expressions. |
rubric |
Plain text that says what a good answer does. |
question |
A question about the final answer. |
expect |
The answer that scores 1 for a yes or no question: “yes” or “no”. |
options |
An object of at least two answers, each with a number from 0 to 1 that says what the answer is worth. |
Tool kinds
Section titled “Tool kinds”A kind is one of these words.
| Kind | Meaning |
|---|---|
shell |
Ran a command. |
read |
Read a file. |
write |
Created or replaced a file. |
edit |
Changed part of a file, or deleted one. |
search |
Searched or listed files. |
web |
Fetched a page or searched the web. |
skill |
Called the harness’s skill tool. |
subagent |
Started another agent. |
other |
Any tool the harness’s adapter has no kind for, such as an MCP tool. |