Changelog
Unreleased
Section titled “Unreleased”- The help text is written from the same tables that parse the command line. It lists the commands, then every flag once with its default and the commands that take it. No command, flag, or default changed.
- The documentation is a website next to the landing page. Its reference pages for the checks, the commands and flags, the config file, the harnesses, and the API are generated from the code, and the tests fail when a committed page or a documented example no longer matches the tool.
- Releases are guarded.
scripts/release-plan.shrefuses a version tag that does not matchpackage.json, is not onmain, has no changelog entry, or is not above the latest version on npm, and theRelease checkworkflow runs the same guards on every pull request. The publish workflow also makes the GitHub release.
The first release. The runner comes from the evals/ folder of magic, where it scored that one set of skills.
-
Run it in any project. The project is the git repository that holds the working directory, and the cases default to
evals/casesin it. -
Score a plugin, a plain folder of skills, or
none, which loads no skills and is the baseline for a comparison. -
Set defaults in an optional
skill-evals.config.json. -
Use it from code with
evaluate,loadCases, andHARNESSES. -
A prompt that starts with a dash reaches the agent in every harness.
-
Windows is not supported.
-
The package is written in TypeScript. It ships compiled JavaScript and type declarations, and still has no dependencies.
-
A
decisioncheck asks a decision model a question about the final answer. The case sets what each allowed answer is worth, the model returns a probability for each answer, and the check’s score is the expected value. All decision checks of a case go out in one request for each run. -
--decision-modeland--decision-url, anddecisionModelanddecisionUrlin the config file, set the model and its endpoint. The client speaks Baseten’s System One API and reads its key fromBASETEN_API_KEY. It was written from the API’s documentation. It has scored checks against bison-1 on a self-hosted deployment, and has not yet called Baseten’s hosted API. -
--decision-timeout <seconds>anddecisionTimeoutin the config file set how long one request to the decision model may take. The default stays 60 seconds. A cold start of a self-hosted model can take that long. -
A decision reply whose probabilities were rounded to 6 decimals is accepted. The sum over the allowed answers may differ from 1 by 0.0000005 for each answer, where the limit was 0.000001 in total, which rejected three honest options that added up to 0.999999.