Skip to content

Read the report

The report has one section per harness and one row per case. run shows each value as A / B (B - A). score shows one value.

  • Score is the case score.
  • Checks shows each check. A deterministic check shows how many valid runs passed it, for example 2/3. A judged check or a decision check shows its mean score, for example 0.67. A decision check starts with the word decision and ends with the answer it expects or its options.
  • Duration, input tokens, output tokens, cost show the median over valid runs. Input tokens include cached tokens. n/a means the harness does not report that number.
  • Suite score is under the table.
  • Judge is one line under the suite score with the number of judge calls and their total duration, tokens, and cost. These are not part of the table’s numbers.
  • Decision model is one more line of the same kind, with the number of requests to the decision model and their total duration and tokens.
  • The first lines name each subject, the harness that ran, and why that harness was chosen.

One repetition tells you little. A check that passes 3 of 3 times on A and 1 of 3 on B is a signal. A check that passes 1 of 1 on A and 0 of 1 on B is not. Judged scores move between runs of the same subject, so read a difference in a judged check against that spread.

run and score print a Markdown report and the path of an output directory. Read the report first, then open the final answers of a case:

<out>/results.json every run record, with check scores, judge reasons, and probabilities
<out>/<harness>/<case>/a1.final.md final answer of subject A, repetition 1
<out>/<harness>/<case>/b1.final.md final answer of subject B, repetition 1
<out>/<harness>/<case>/a1.stream.jsonl the harness's raw event stream for that run

score names its files subject1.final.md and so on.