Read the report
The report has one section per harness and one row per case. run shows each value as A / B (B - A). score shows one value.
- Score is the case score.
- Checks shows each check. A deterministic check shows how many valid runs passed it, for example
2/3. A judged check or a decision check shows its mean score, for example0.67. A decision check starts with the worddecisionand ends with the answer it expects or its options. - Duration, input tokens, output tokens, cost show the median over valid runs. Input tokens include cached tokens.
n/ameans the harness does not report that number. - Suite score is under the table.
- Judge is one line under the suite score with the number of judge calls and their total duration, tokens, and cost. These are not part of the table’s numbers.
- Decision model is one more line of the same kind, with the number of requests to the decision model and their total duration and tokens.
- The first lines name each subject, the harness that ran, and why that harness was chosen.
One repetition tells you little. A check that passes 3 of 3 times on A and 1 of 3 on B is a signal. A check that passes 1 of 1 on A and 0 of 1 on B is not. Judged scores move between runs of the same subject, so read a difference in a judged check against that spread.
The output directory
Section titled “The output directory”run and score print a Markdown report and the path of an output directory. Read the report first, then open the final answers of a case:
<out>/results.json every run record, with check scores, judge reasons, and probabilities<out>/<harness>/<case>/a1.final.md final answer of subject A, repetition 1<out>/<harness>/<case>/b1.final.md final answer of subject B, repetition 1<out>/<harness>/<case>/a1.stream.jsonl the harness's raw event stream for that runscore names its files subject1.final.md and so on.