# Autobench

> Autobench is a YAML-first Python framework for collecting semantic, replayable benchmark evidence from AI and non-AI systems. It provides datasets, variants, tasks, instrumentation, scoring, derivation, immutable records, replay, reporting, and tracked asset lineage without making OpenTelemetry or a specific evaluation runtime part of its core.

Use this index to select the smallest relevant documentation page. Use the full bundle when broad context is required.

## Start

- [Getting Started](https://vcoderun.github.io/autobench/_markdown/getting-started.txt): Install Autobench, define a first YAML benchmark, validate it, run it, and replay the recorded experiment.
- [Examples](https://vcoderun.github.io/autobench/_markdown/examples.txt): Offline and live benchmark, ABP instrumentation, replay, and CodeMode integrations.
- [Complete Feature Guide](https://vcoderun.github.io/autobench/_markdown/current-capabilities-guide.txt): End-to-end guide to the complete supported feature set.
- [Full Documentation](https://vcoderun.github.io/autobench/llms-full.txt): All public documentation pages concatenated as Markdown.

## Foundations

- [Capability Map](https://vcoderun.github.io/autobench/_markdown/capabilities.txt): Ownership boundaries and a complete capability inventory.
- [Core Concepts](https://vcoderun.github.io/autobench/_markdown/concepts.txt): Benchmark, case, variant, factor, metric, observation, span, record, and asset terminology.
- [Datasets and Variants](https://vcoderun.github.io/autobench/_markdown/datasets-and-variants.txt): Inline and file-backed cases, matrix expansion, factor semantics, and identity rules.
- [Tasks and Runtime](https://vcoderun.github.io/autobench/_markdown/tasks-and-runtime.txt): Callable resolution, sync and async tasks, run context, lifecycle, concurrency, retries, and failures.
- [Observations and Semantics](https://vcoderun.github.io/autobench/_markdown/observations-and-semantics.txt): Semantic types, projections, queries, duplicate resolution, provenance, and typed evidence.

## Workflows

- [Instrumentation and Traces](https://vcoderun.github.io/autobench/_markdown/instrumentation-and-traces.txt): Manual spans, method instrumentation, extractors, trace attachment, and Pydantic AI usage capture.
- [Native Instrumentation](https://vcoderun.github.io/autobench/_markdown/native-instrumentation.txt): Typed/YAML Pydantic AI, OpenAI, OpenAI Agents, and HTTPX instrumentation, privacy, diagnostics, and layering.
- [Asset Tracking](https://vcoderun.github.io/autobench/_markdown/asset-tracking.txt): Prompt, tool, Pydantic model, dataclass, and class versioning with hashes, history, and diffs.
- [Agentic Evaluation](https://vcoderun.github.io/autobench/_markdown/agentic-evaluation.txt): Tool-call, argument, plan, policy, and trace evaluation primitives.
- [Scoring and Derivation](https://vcoderun.github.io/autobench/_markdown/scoring-and-derivation.txt): Built-in and Python scorers, derived metrics, pricing, policies, repeated measurement, and paired baselines.
- [Recording and Reporting](https://vcoderun.github.io/autobench/_markdown/recording-and-reporting.txt): Immutable RunRecords, artifacts, replay, comparison, leaderboard, matrix, distribution, and export behavior.
- [CLI](https://vcoderun.github.io/autobench/_markdown/cli.txt): Validate, run, replay, report, compare, export, and ABP instrumentation diagnostics.

## Reference

- [YAML Specification](https://vcoderun.github.io/autobench/_markdown/yaml-spec.txt): Complete YAML-first benchmark DSL and resolution rules.
- [Python API](https://vcoderun.github.io/autobench/_markdown/python-api.txt): Typed builder, runtime, instrumentation, recording, replay, and reporting APIs.
- [API Reference](https://vcoderun.github.io/autobench/_markdown/api-reference.txt): Generated public Python API signatures and members.
- [Development](https://vcoderun.github.io/autobench/_markdown/development.txt): Repository setup, quality gates, and contribution workflow.
- [Release Notes 0.2.0](https://vcoderun.github.io/autobench/_markdown/release-notes/0.2.0.txt): ABP, native instrumentation, compatibility, and stability guarantees.
- [Release Notes 0.1.0](https://vcoderun.github.io/autobench/_markdown/release-notes/0.1.0.txt): Initial release scope and guarantees.

## Optional

- [GitHub Repository](https://github.com/vcoderun/autobench): Source code, issues, and releases.
- [Changelog](https://github.com/vcoderun/autobench/blob/main/CHANGELOG.md): Unreleased and released changes.