Skip to content

What was verified

skill-evals check is how you confirm an adapter on your own machine. These tables say what was confirmed when the adapters were written, and how. Only Claude Code ran a real model. “Scripted endpoint” means that the real CLI ran through this runner against a local model endpoint that replays a fixed tool call and a fixed answer, with every request it received logged.

Loading a plugin, confirmed in the magic repository where this runner started, and run again from this package on 2026-11-21:

Harness Version Confirmed Not confirmed
claude-code 2.1.329, 2.1.330 Real runs with a real model: check, run with a git ref, and score. A globally installed copy of the plugin did not load. The parser fixture is a captured stream. A subscription login. The machine used authenticates through environment variables.
pi 0.73.1 Scripted endpoint: the command line is accepted, the plugin’s extension and skills load from the directory, a decoy skill planted in the real home stays out, and the stream parses. The parser fixtures are captured streams. A run with a real model. Cost for a priced model (the scripted model has no price).
opencode 1.19.6 Scripted endpoint: the plugin loads from the config entry, its skills are listed from the directory, a decoy skill stays out, and a native skill call parses. A run with a real model. OpenCode V2, which reads the plugins key.
codex 0.171.0 Scripted endpoint: both install commands succeed in the new home, the plugin’s skills are listed from the plugin cache, a decoy skill stays out, and a command that reads a SKILL.md parses. A run with a real model.
cursor 2026.11.19 The CLI accepts the adapter’s command line and stops at its login check. check reports that as a failed row and goes on. Everything after login. The parser and its fixture come from Cursor’s output format reference, not from a captured run. Whether --plugin-dir loads the plugin’s skills and hooks in print mode.

Loading a skills folder, the none baseline, and a prompt that starts with a dash, confirmed on 2026-11-21:

Harness Version Skills folder Prompt that starts with a dash Not confirmed
claude-code 2.1.330 Real runs with a real model: run --a none --b <folder> gave 0 of 2 skill loads without the folder and 2 of 2 with it. The harness listed the skill at startup, so a folder that does not load makes the run invalid. A skill of the same name in the real home’s ~/.claude/skills did not load. A real run: after -- the prompt reached the model. Without -- the CLI rejects it as an unknown option. A skill whose front matter name differs from its folder.
pi 0.73.1 Scripted endpoint: the copy named with --skill is in the model request with discovery off, and decoy skills in the real home’s ~/.pi/agent/skills and ~/.agents/skills are not. Scripted endpoint: the prompt on standard input is the user message, unchanged. Pi has no -- separator and rejects such a prompt as an argument. A run with a real model.
opencode 1.19.6 Scripted endpoint: the copy in ~/.agents/skills is in the model request, and the decoys are not. Scripted endpoint: after -- the prompt is the user message. Without -- the CLI prints its help. A run with a real model.
codex 0.171.0 Scripted endpoint: the model request lists ~/.agents/skills of the new home as a skill root with the copy in it, and the decoys are not there. Scripted endpoint: after -- the prompt is the user message. Without -- the CLI rejects it. A run with a real model.
cursor 2026.11.19 The installed bundle’s code has .cursor/skills under the home directory as a place it reads skills from. No run confirmed it. After -- the CLI gets past its option parser and stops at its login check. Without -- it rejects the prompt. Everything after login, so whether the skill and the prompt reach the model.

With none, the scripted requests of Pi, OpenCode, and Codex held neither the subject’s skill nor a decoy.

A prompt that names a skill invokes it, confirmed in Claude Code 2.1.330 with a real model on 2026-11-21: with a skills folder loaded, Use the greet skill to give me a greeting to start my day. loaded the skill in 2 of 2 runs, and the answer held the sentence that the skill asks for in 2 of 2. With none, 1 of the 2 runs still called the skill tool with that name, which counts as a load. The same was not run in the other four harnesses.

Tool kinds and the judge:

Harness Tool kinds Judge run with nothing loaded
claude-code Skill, Bash, Read, Edit, Write, WebFetch, and Agent come from a captured run, which is a parser fixture. Grep, Glob, WebSearch, NotebookEdit, and Task are mapped but were not seen in a stream. Version 2.1.330 searched through Bash. Real runs.
pi The seven tool names come from the installed package (0.73.1). read was seen in a captured stream. The same command line as the none baseline, which ran against the scripted endpoint. No judge reply was parsed.
opencode The names come from the installed binary (1.19.6). skill and read were seen in a stream. As for Pi.
codex The item types come from the Codex source for exec --json events. command_execution was seen in a captured stream. As for Pi.
cursor The names come from the installed bundle (2026.11.19). read and write are in the documented example stream. No run confirmed them. Not run.

The decision model:

Part Confirmed Not confirmed
Baseten System One client Written from the System One API reference and the decision models guide as they were on 2026-11-21. The tests run the client against a local HTTP server that answers in the documented response shape: the request body, the Authorization header, the scoring, the retry, and every unusable reply. The client was also run against bison-1, self-hosted on Modal with the runtime that Baseten publishes for its sibling models. Yes, no, and options checks scored, the options with labels that hold spaces. The reply reported token usage. A warm request took about 1.2 seconds. Repeated gradings of a fixed answer gave identical scores. Baseten’s hosted API itself, including its auth scheme. No request from this client has reached it.

Two limits apply across harnesses. A login stored in the operating system’s keychain may not follow the new home. If check reports a login error, set the harness’s API key variable. Only Claude Code reports what it loaded, as What one run does says.