Skip to content

Checks

This page is generated from src/checks.ts, so it says what the tool does.

A check is one object in the checks list of a case.json. Every check has a type, and takes an optional weight, a positive number that defaults to 1. Write a case shows how to use them.

Check Fields Scores 1 when
skill_loaded skill The agent loaded that skill.
skill_not_loaded skill The agent did not load that skill.
output_matches pattern, optional flags The final answer matches the regular expression.
output_lacks pattern, optional flags The final answer does not match the regular expression.
output_contains text The final answer contains that text.
output_not_contains text The final answer does not contain that text.
file_exists path That file exists in the workspace after the run.
file_matches path, pattern, optional flags That file exists and matches the regular expression.
tool_called at least one of kind, name, optional inputMatches, optional flags The agent made a tool call that fits every given field.
tool_not_called at least one of kind, name, optional inputMatches, optional flags The agent made no tool call that fits every given field.
judge rubric Not 0 or 1. A model scores the final answer against the rubric.
decision question, optional expect. It takes exactly one of expect and options. Not 0 or 1. A decision model gives each allowed answer a probability, and the score is the expected value.
Field Holds
skill The name of a skill.
pattern A regular expression.
text Text to look for. The match is exact and case-sensitive.
path A file path inside the workspace.
kind What the tool does, with the same word in every harness.
name A regular expression for the harness’s own tool name.
inputMatches A regular expression for the tool call’s input, written as JSON.
flags The flags of the check’s regular expressions.
rubric Plain text that says what a good answer does.
question A question about the final answer.
expect The answer that scores 1 for a yes or no question: “yes” or “no”.
options An object of at least two answers, each with a number from 0 to 1 that says what the answer is worth.

A kind is one of these words.

Kind Meaning
shell Ran a command.
read Read a file.
write Created or replaced a file.
edit Changed part of a file, or deleted one.
search Searched or listed files.
web Fetched a page or searched the web.
skill Called the harness’s skill tool.
subagent Started another agent.
other Any tool the harness’s adapter has no kind for, such as an MCP tool.