CLI¶

The CLI is human-first: validation, runs, replay, reports, and comparisons render Rich panels and tables. YAML, CSV, and Markdown are explicit file exports rather than raw terminal dumps.

Command Summary¶

References inside a benchmark spec resolve relative to the spec file.

Commands¶

Validate¶

uv run autobench validate path/to/spec.yaml

Validation loads external datasets and referenced configuration, checks duplicate IDs and task requirements, resolves source files, and renders case, variant, and planned-run counts. It does not execute the task target.

Run¶

uv run autobench run path/to/spec.yaml --record runs/example

Options:

• --concurrency INTEGER: maximum active runs, default 1, minimum 1.
• --record DIRECTORY: explicit immutable record directory.
• --no-record: execute and report without persistence.

Without either recording flag, Autobench writes under .autobench/<spec-name>/<experiment-id>/.

Replay¶

uv run autobench replay runs/example

Replay does not import tasks, scorers, or application modules. It reconstructs the experiment from the record directory and renders the recorded report configuration.

Report¶

uv run autobench report runs/example

report emphasizes status, variant configuration, leaderboards, per-run metrics, case matrices, comparisons, and distributions.

Export¶

uv run autobench export runs/example --format yaml --path runs/example/report.yaml
uv run autobench export runs/example --format csv --path runs/example/runs.csv
uv run autobench export runs/example --format markdown --path runs/example/report.md

Compare¶

uv run autobench compare runs/example --baseline baseline --candidate optimized

Both IDs must exist in the recorded experiment. The command shows paired-run count, changed factors, aggregate metric deltas, and a confounding flag.

Instrumentation Diagnostics¶

uv run autobench instrumentation doctor
uv run autobench instrumentation trace runs/example

doctor inspects every built-in integration without importing unavailable SDKs. Its Rich tables show target versions, supported status, abstraction layer, hook/patch mechanism, sync/async/streaming support, span and semantic families, capture defaults, extras, and degradation details.

trace operates only on recorded evidence. It reports case/variant span counts, roots, partial state, diagnostics, span kinds, and instrumentor composition without importing the benchmark task or provider SDKs.

CLI Behavior¶

• run executes the benchmark matrix, optionally records it, and renders Rich summary tables.
• replay, report, export, and compare operate on recorded evidence.
• instrumentation trace has the same replay-only dependency boundary.
• report and compare render Rich terminal views instead of dumping Markdown or YAML.
• export always writes a file and then shows a Rich preview of the exported projection.
• default recording paths are placed under .autobench/<spec-name>/<experiment-id>/.

Exit And Error Behavior¶

• Invalid YAML, schema errors, unresolved tasks, recording collisions, and missing records return a nonzero exit code.
• User-facing errors include the relevant file and YAML location when available.
• A process may complete while individual runs are failed, errored, or skipped; the status table makes those states explicit.
• Replay and reporting never fall back to live benchmark execution.

Typical Workflow¶

autobench validate autobench.yaml
autobench run autobench.yaml --concurrency 4 --record runs/candidate-42
autobench report runs/candidate-42
autobench compare runs/candidate-42 --baseline baseline --candidate candidate
autobench export runs/candidate-42 --format csv --path analysis/runs.csv