# Autobench Full Documentation > Complete, LLM-readable documentation for Autobench: a YAML-first framework for semantic, replayable benchmark evidence. Canonical documentation: https://vcoderun.github.io/autobench/ This file follows the site navigation order. Each section includes the canonical page URL followed by its complete Markdown source. --- ## Home Canonical page: https://vcoderun.github.io/autobench/ # Autobench **Turn one-off benchmark scripts into replayable semantic experiment data.** Autobench is a YAML-first benchmark and evidence framework. It runs deterministic case and variant matrices, records typed observations and artifacts, and lets you replay or compare the evidence without executing the subject again. ```bash uv add autobench autobench validate autobench.yaml autobench run autobench.yaml --record runs/latest ``` The core loop is: ```text Dataset / Cases x Variants / Factors -> Task executes the subject -> Context and spans collect observations -> Scorers evaluate outputs -> Derivers add semantic metrics -> Recorder writes immutable YAML evidence -> Replay, report, export, and compare operate on recorded runs ``` Autobench is designed for AI-heavy systems, but the runtime itself is generic. If you can express a case, a variant, a task, and semantic outcomes, Autobench can benchmark it. ## Why It Exists Most benchmark codebases keep re-implementing the same machinery: - scenario loading - case x variant expansion - task orchestration - metrics and derived metrics - artifacts and replay - comparison and reporting Autobench provides those utilities as framework primitives so users describe the benchmark instead of rebuilding the runner. ## What Autobench Owns Autobench is more than a matrix runner. It owns the evidence lifecycle from benchmark definition to optimization-ready records: | Layer | Capabilities | | --- | --- | | Definition | YAML DSL, Python builder, datasets, case defaults, variants, factors, schema hints | | Execution | deterministic matrix planning, sync/async tasks, concurrency, failure isolation, progress events | | Evidence | semantic observations, spans, artifacts, checks, diagnostics, errors, trace envelopes | | Evaluation | six scorer kinds, expected-action evaluation, policies, metric packs, custom scorers | | Derivation | token cost, tiered pricing, paired baselines, verdicts, measurement statistics | | Lineage | prompt/tool/type/config tracking, structured schemas, source hashes, versions, diffs | | Persistence | immutable YAML RunRecords, source hashes, environment metadata, portable artifacts | | Analysis | replay, Rich reports, leaderboards, case matrices, comparisons, distributions, exports | | Optimization | compact feedback records and semantic evidence for pydantic-gepa and autoptimize | See the [Capability Map](capabilities.md) for the complete feature inventory and ownership boundaries. ## Choose A Path | Goal | Start here | | --- | --- | | Run the smallest complete benchmark | [Getting Started](getting-started.md) | | See everything Autobench supports | [Capability Map](capabilities.md) | | Learn the evidence model | [Core Concepts](concepts.md) | | Adapt a working integration | [Examples](examples.md) | | Define a benchmark declaratively | [YAML Spec](yaml-spec.md) | | Instrument an existing application | [Instrumentation And Traces](instrumentation-and-traces.md) | | Track prompts, tools, and schemas | [Asset Tracking](asset-tracking.md) | | Evaluate agent behavior | [Agentic Evaluation](agentic-evaluation.md) | | Replay and compare recorded evidence | [Recording And Reporting](recording-and-reporting.md) | ## Start Here - [Getting Started](getting-started.md) - [Capability Map](capabilities.md) - [Examples](examples.md) - [YAML Spec](yaml-spec.md) - [Python API](python-api.md) --- ## Getting Started Canonical page: https://vcoderun.github.io/autobench/getting-started/ # Getting Started ## Install Install the released package in an application: ```bash uv add autobench ``` For repository development and the bundled examples: ```bash uv sync --extra dev ``` ## Validate A Spec Validate the smallest complete benchmark: ```bash uv run autobench validate examples/minimal/autobench.yaml ``` Expected output includes: - benchmark id - case count - variant count - planned run count ## Run And Record ```bash uv run autobench run examples/minimal/autobench.yaml --record /tmp/autobench-minimal ``` This executes the task matrix and writes: - `experiment.yaml` - `summary.yaml` - per-run `cases///run.yaml` - artifact payloads under `artifacts/` ## Replay, Report, Export ```bash uv run autobench replay /tmp/autobench-minimal uv run autobench report /tmp/autobench-minimal uv run autobench export /tmp/autobench-minimal --format yaml --path /tmp/minimal-report.yaml uv run autobench export /tmp/autobench-minimal --format csv --path /tmp/minimal-runs.csv ``` Replay does not import or execute the original task target. It only reads recorded evidence. Report and compare render Rich terminal tables. Export writes a file and shows a Rich preview. ## Quality Gates ```bash make prod make pre-commit make docs make examples ``` `make prod` covers tests, `100%` source line and branch coverage, formatting checks, linting, typing, docs, the Python validation matrix, and all offline examples. Preview the documentation while editing: ```bash make docs-serve ``` --- ## Examples Canonical page: https://vcoderun.github.io/autobench/examples/ # Examples The release examples are applications of the public framework, not alternate runtimes or mock-only snippets. Every offline example executes the complete `run -> record -> replay -> report -> export` workflow through `make examples`. ## Minimal ```bash uv run autobench run examples/minimal/autobench.yaml --record /tmp/autobench-minimal ``` Demonstrates inline cases, deterministic variants, exact scoring, a case matrix, and comparison. ## Basic ```bash uv run autobench run examples/basic/autobench.yaml --record /tmp/autobench-basic ``` Routes file-backed support tickets and records workflow spans plus decision artifacts. The second variant fixes an enterprise-outage routing failure, making the comparison visible in terminal tables. ## Mid ```bash uv run autobench run examples/mid/autobench.yaml --record /tmp/autobench-mid ``` Records semantic token usage and latency, derives request cost from a local pricing DSL, applies success and cost policies, and renders cost distributions. ## Advanced ```bash uv run autobench run examples/advanced/autobench.yaml --record /tmp/autobench-advanced ``` Uses repeated measurements and sample artifacts, then derives per-case speedup with a paired baseline. Correctness remains a constraint while speed is the optimization objective. ## CodeMode ```bash uv run python examples/codemode/run_benchmark.py --only parse_cron ``` This is a live integration with the runtime that provides `vowel.codemode`. It generates evaluation specs with configured models, replays each generated spec against the source function, and records coverage, latency, generated specs, and exploration artifacts as Autobench evidence. It requires the external CodeMode runtime, an `OPENROUTER_API_KEY`, and network access. CodeMode-specific calls remain in the example task. Autobench core only owns the generic dataset, variant, task, observation, artifact, scoring, recording, replay, and reporting seams. ## Pydantic AI ```bash uv sync --extra instrumentation export OPENROUTER_API_KEY=... export OPENROUTER_MODEL=openrouter:openai/gpt-5.6-luna uv run python examples/pydantic_ai/openrouter_instrument_all.py \ --record /tmp/autobench-openrouter ``` This live benchmark makes a real OpenRouter request through Pydantic AI's OpenAI-compatible model. It uses a tracked prompt, a catalog tool, streaming execution, and structured Pydantic output. `Benchmark.instrument_all()` discovers Pydantic AI, OpenAI, and HTTPX automatically, producing a layered framework/client/transport trace plus semantic token, model, latency, tool, validation, streaming, HTTP, scoring, and asset-version evidence. The task contains no manual `ctx.span()` or `ctx.metric()` calls. The complete experiment is recorded as replayable YAML under the supplied directory. `agent_benchmark.py` remains the provider-neutral variant for any configured Pydantic AI model. ## ABP Manual And Method ```bash uv run autobench run examples/abp_manual/autobench.yaml --record /tmp/abp-manual ``` Combines a manual workflow span with automatic `TicketRouter.route` instrumentation. The method instrumentor emits a nested span, one metric, and one factor while preserving the method signature and result. ## ABP Concurrent ```bash uv run autobench run examples/abp_concurrent/autobench.yaml --concurrency 2 \ --record /tmp/abp-concurrent ``` Runs asynchronous worker siblings under a workflow span. It demonstrates task-local ABP context, correct sibling parentage, and concurrent benchmark matrix execution. ## OpenAI Streaming ```bash uv sync --extra instrumentation uv run python examples/abp_openai/run_openai_streaming.py ``` Uses the official OpenAI client with a real streaming parser and an offline HTTPX mock transport. The resulting trace contains client and transport spans, first-chunk evidence, and normal stream completion without network access or credentials. ## OpenAI Agents ```bash uv sync --extra openai-agents uv run python examples/abp_openai_agents/run_openai_agents.py ``` Runs a real OpenAI Agents workflow, function span, and custom span through the native trace processor. No model or network call is required. ## Replay And Extraction ```bash uv run python examples/abp_replay/replay_and_extract.py /tmp/recorded-experiment ``` Loads each recorded RunRecord and creates an immutable extraction-derived record with signal, span, and usage observations. The script imports no provider SDK. --- ## Complete Feature Guide Canonical page: https://vcoderun.github.io/autobench/current-capabilities-guide/ # Autobench Su An Ne Yapiyor? Bu dosya Autobench'i hic bilmeyen birine anlatmak icin yazildi. Amac kisa bir README yazmak degil; Autobench'in bugunku halinin hangi problemi cozdunu, hangi parcalardan olustugunu, hangi API'lerin nasil kullanildigini ve bir uygulama benchmark'inin nasil Autobench'e tasinacagini tek dosyada, bol kod ornegiyle gostermek. Autobench su anda bir "benchmark script generator" degil. Daha dogru tanim su: > Autobench, uygulama veya AI sistemi calistirirken ortaya cikan benchmark > kanitlarini semantic, replayable ve export edilebilir bir deney kaydina > donusturen YAML-first bir evidence framework'tur. Bu cumledeki kelimeler onemli: - **YAML-first:** Benchmark tanimi oncelikle insan tarafindan okunabilir bir YAML spec olarak dusunulur. - **Evidence:** Sadece "score 0.82" yazmaz; case, variant, factor, metric, span, artifact, error, asset version ve environment bilgisini beraber saklar. - **Semantic:** Metrikler sadece `score`, `cost`, `tokens` gibi rastgele isimler degildir. `quality.score`, `money.cost`, `llm.tokens.input`, `time.latency`, `agent.tool.argument.correctness` gibi anlam siniflarina baglanabilir. - **Replayable:** Benchmark calistiktan sonra task'i tekrar import etmeden ve modele tekrar istek atmadan rapor, compare ve export uretilebilir. - **General-purpose:** AI/agent benchmark'lari birinci sinif use case, ama framework sadece LLM eval icin yazilmadi. Latency, throughput, correctness, cost, domain KPI, tool kalitesi, policy compliance gibi her tur sistemi olcebilir. Su an v0.1 cizgisinde olan seylerin ozeti: - YAML benchmark spec okuma ve dogrulama - Dataset, case, case defaults, attachment ve file-backed dataset destegi - Case x variant matrisini deterministik kosma - Python task runtime: sync ve async task destegi - `RunContext` ile metric, factor, event, diagnostic, check, outcome, artifact ve span toplama - Span duration'i framework tarafindan hesaplama - Manual measurement helper: warmup, repetition, max_seconds, median, p95, noise vb. - Scoring: output, pass/fail, exact, schema, python callback, expected action - Semantic registry ve semantic alias/parent sistemi - Token cost derivation: token + provider + model -> `money.cost` - Paired baseline post-derivation: baseline/candidate run'larini case veya factor bazinda eslestirip speedup/delta uretme - Policy checks: `must_greater_equal`, `must_less_equal`, `must_between`, vb. - Immutable YAML RunRecord ve ExperimentRecord yazma - Replay: task import etmeden recorded evidence'dan `ExperimentResult` kurma - Rich CLI tablolar: validate, run, replay, report, export, compare - Markdown/YAML/CSV export - Asset tracking: prompt, tool, Pydantic model, dataclass, typed class, field/param schema, version hash, source hash, asset history - Pydantic Evals payload adapter - Pydantic AI usage bridge - Trace envelope import - Agentic metric primitives: agent/tool span'lari, expected tool action scoring - Production sample -> case ve generated/synthetic case metadata helper'lari Su an bilincli olarak v0.1 yuzeyinden cikarilmis seyler: - Chart/image export - Dashboard - Hosted platform - Full OTel bridge - Full autoptimize implementation - Causal attribution iddiasi Autobench bugun chart engine degil. Rich terminal tablolar, Markdown, YAML ve CSV export var; chart/image artifact tasarimi v1 sonrasi icin bekletiliyor. --- ## 1. Neden Autobench Var? Bir benchmark scripti genelde boyle baslar: ```python # run_benchmark.py for scenario in scenarios: for model in models: started = perf_counter() result = run_my_app(scenario, model) elapsed = perf_counter() - started score = judge(result) rows.append( { "scenario": scenario.name, "model": model, "score": score, "latency": elapsed, "tokens": result.usage.total_tokens, } ) write_json(rows, "summary.json") print_table(rows) ``` Bu pratikte calisir ama hizla dagilir: - Case'ler nerede? - Hangi variant hangi factor'leri degistirdi? - Metriklerin semantic anlami ne? - Token input/output ayri mi tutuldu? - Cost nasil hesaplandi? - Hangi prompt veya tool version'i kullanildi? - Bir run tekrar raporlanabilir mi? - Task import etmeden replay yapilabilir mi? - Baseline/candidate farki case bazinda mi aggregate bazinda mi? - Score mu metric mi, objective mi diagnostic mi? - Bir policy fail olursa run status nasil etkilenir? - Degisen tool/prompt/schema benchmark sonucunu nasil etkiledi? Autobench bu sorulari framework primitive'lerine ayirir: ```text BenchmarkSpec -> Dataset / Case -> Variant / Factor -> Task -> RunContext -> Span / Observation / Artifact -> Scorer -> Deriver / PostDeriver / Policy -> RunRecord -> ExperimentRecord -> Replay / Report / Export / Compare ``` Yani benchmark yazarken tekrar tekrar ayni runner iskeletini yazmak yerine, deneyi tanimlar ve framework'un evidence pipeline'ina teslim edersin. --- ## 2. En Kucuk Mental Model Autobench'i anlamak icin su bes kavram yetiyor: ### Case Bir test girdisi. Ornegin bir destek talebi: ```python from autobench import Case case = Case( id="refund_001", input={ "subject": "Duplicate charge", "body": "I was billed twice yesterday.", }, expected={ "queue": "billing", "priority": "high", }, tags=["billing", "refund"], ) ``` ### Variant Ayni case'i farkli konfigurasyonlarla kosmak icin kullanilir. Ornegin prompt version veya model degisikligi: ```python from autobench import FactorValue, Semantic, Variant variant = Variant( id="route_v2", label="new routing prompt", factors=[ FactorValue( name="prompt_version", value="route-v2", semantic_type=Semantic.PROMPT_VERSION, optimize=True, ), FactorValue( name="model", value="openrouter:openai/gpt-5.6-luna", semantic_type=Semantic.LLM_MODEL_NAME, optimize=True, ), ], ) ``` ### Task Autobench'in calistirdigi uygulama fonksiyonu. Signature kuralidir: ```python def run_case(ctx, case): ... ``` `ctx` her zaman birinci parametre, `case` her zaman ikinci parametredir. ### Observation Benchmark sirasinda toplanan her anlamli veri parcasidir: - metric: `quality.score`, `time.latency`, `money.cost` - factor: `llm.model.name`, `prompt.version` - event: diagnostic, error, skip reason - artifact reference: trace, raw response, generated file ### RunRecord Bir case x variant kosusunun immutable kaydidir. Task output, observations, scores, spans, artifacts, errors, factors ve asset versions bu kayda girer. --- ## 3. Kurulum Ve CLI Repository icinde gelistirme kurulumu: ```bash uv sync --extra dev ``` Autobench CLI komutlari: ```bash uv run autobench --help ``` Mevcut komutlar: ```text validate Validate a YAML Autobench spec. run Run a YAML Autobench spec. replay Replay recorded YAML evidence without importing benchmark tasks. report Render the Rich terminal report from recorded evidence. export Export recorded evidence to a file and show a Rich terminal preview. compare Compare two recorded variants without claiming causality. ``` Tipik workflow: ```bash uv run autobench validate autobench.yaml uv run autobench run autobench.yaml --record runs/support-routing uv run autobench replay runs/support-routing uv run autobench report runs/support-routing uv run autobench compare runs/support-routing --baseline route_v1 --candidate route_v2 uv run autobench export runs/support-routing --format yaml --path runs/support-routing/report.yaml uv run autobench export runs/support-routing --format markdown --path runs/support-routing/report.md uv run autobench export runs/support-routing --format csv --path runs/support-routing/runs.csv ``` `run` icin concurrency: ```bash uv run autobench run autobench.yaml --record runs/demo --concurrency 4 ``` Record istemiyorsan: ```bash uv run autobench run autobench.yaml --no-record ``` Export formatlari su anda: ```text csv markdown yaml ``` Chart/image export v0.1 yuzeyinde yoktur. --- ## 4. Ilk Tam Ornek: Support Routing Benchmark Diyelim ki bir support ticket router'in var. Input olarak ticket aliyor, output olarak hangi kuyruya gitmesi gerektigini soyluyor. ### 4.1 Uygulama kodu `app/router.py`: ```python from __future__ import annotations def route_ticket(ticket: dict[str, str], *, prompt_version: str) -> dict[str, object]: text = f"{ticket.get('subject', '')} {ticket.get('body', '')}".lower() if "refund" in text or "billed" in text or "charge" in text: queue = "billing" confidence = 0.94 if prompt_version == "route-v2" else 0.86 elif "password" in text or "login" in text: queue = "account" confidence = 0.88 else: queue = "general" confidence = 0.72 return { "queue": queue, "confidence": confidence, "matched": confidence >= 0.8, } ``` ### 4.2 Benchmark task'i `app/benchmarks/support.py`: ```python from __future__ import annotations from autobench import DurationMetricSpec, Semantic, SpanKind from app.router import route_ticket def run_ticket_case(ctx, case): prompt_version = ctx.factor("prompt_version") with ctx.span( "route_ticket", kind=SpanKind.WORKFLOW, input=case.input, attributes={"prompt_version": prompt_version}, duration_metric=DurationMetricSpec(name="routing_duration"), ) as span: output = route_ticket(case.input, prompt_version=prompt_version) span.set_output(output) span.factor( "prompt_version", prompt_version, semantic_type=Semantic.PROMPT_VERSION, ) span.metric( "confidence", output["confidence"], semantic_type=Semantic.QUALITY_SCORE, ) span.outcome(output["matched"]) return output ``` Burada dikkat edilecek noktalar: - `ctx.factor("prompt_version")` aktif variant'tan factor okur. - `ctx.span(...)` benchmark icindeki anlamli bir is parcasini kaydeder. - `duration_metric=...` verdigin icin span bitince latency otomatik metric olur. - `span.metric`, `span.factor`, `span.outcome` observation yazar. - Return edilen `output`, scorer'lar tarafindan okunabilir. ### 4.3 Dataset `datasets/support_cases.yaml`: ```yaml cases: - id: refund_request input: subject: Duplicate card charge body: I was billed twice yesterday and need a refund. expected: queue: billing tags: [billing, refund] - id: login_problem input: subject: Cannot login body: My password reset link expired. expected: queue: account tags: [account] ``` ### 4.4 Benchmark spec `autobench.yaml`: ```yaml benchmark: support-routing: description: Route support tickets into the right queue. cases: datasets/support_cases.yaml run: python: app.benchmarks.support:run_ticket_case variants: route_v1: label: baseline routing prompt factors: prompt_version: value: route-v1 semantic: prompt.version optimize: true route_v2: label: improved routing prompt factors: prompt_version: value: route-v2 semantic: prompt.version optimize: true score: matched: pass: output.matched semantic: result.success role: objective goal: maximize queue_correct: exact: actual: output.queue expected: case.expected.queue semantic: quality.correctness role: objective goal: maximize confidence: value: output.confidence semantic: quality.score role: diagnostic goal: maximize report: leaderboard: show: pass_rate: metric: result.success aggregate: ratio_true avg_correctness: metric: quality.correctness aggregate: mean avg_confidence: metric: quality.score aggregate: mean matrix: metric: quality.correctness compare: route_v1 -> route_v2: show: correctness_delta: metric: quality.correctness aggregate: mean ``` ### 4.5 Calistirma ```bash uv run autobench validate autobench.yaml uv run autobench run autobench.yaml --record runs/support-routing uv run autobench report runs/support-routing uv run autobench export runs/support-routing --format markdown --path runs/support-routing/report.md ``` Bu noktada Autobench sunlari yapar: 1. YAML spec'i parse eder. 2. Dataset'i yukler. 3. Variant'lari normalize eder. 4. Case x variant matrisini kurar. 5. Her run icin `RunContext` yaratir. 6. `app.benchmarks.support:run_ticket_case` fonksiyonunu cagirir. 7. Task observation'larini toplar. 8. Scorer'lari kosar. 9. Score'lari metric observation'a cevirir. 10. Policy/derivation varsa uygular. 11. RunRecord ve ExperimentRecord yazar. 12. Rich terminal tablolarla ozet gosterir. --- ## 5. YAML Spec Tasarimi Autobench YAML DSL iki formu destekler: 1. Daha insan dostu authoring DSL. 2. Daha structured internal payload. Yeni benchmark yazarken authoring DSL daha okunaklidir: ```yaml benchmark: my-benchmark-id: description: Human readable benchmark description. cases: datasets/cases.yaml run: python: my_package.benchmarks:run_case variants: baseline: factors: model: value: openrouter:openai/gpt-5.6-luna semantic: llm.model.name optimize: true score: success: pass: output.ok semantic: result.success report: leaderboard: show: success_rate: metric: result.success aggregate: ratio_true ``` Autobench tarafindan yazilan YAML'lar schema header alir. Schema cache path'i Autobench versiyonuna baglidir: ```yaml # yaml-language-server: $schema=/Users/you/.autobench/0.2.0/schemas/benchmark_schema.json ``` Bu sayede editor tarafinda auto-completion hedeflenir. ### 5.1 Dataset inline yazilabilir ```yaml benchmark: inline-demo: dataset: cases: - id: case_1 input: value: 10 expected: doubled: 20 run: python: app.tasks:double variants: default: {} ``` ### 5.2 Dataset dosyadan okunabilir ```yaml benchmark: file-backed-demo: cases: datasets/cases.yaml run: python: app.tasks:run_case ``` `datasets/cases.yaml`: ```yaml cases: - id: case_1 input: value: 10 expected: doubled: 20 ``` ### 5.3 Case defaults Case defaults ortak metadata, tags veya input/expected parcasi vermek icin kullanilir: ```yaml benchmark: defaulted-demo: dataset: defaults: metadata: owner: support-team tags: [smoke] input: locale: tr-TR cases: - id: refund_tr input: subject: Para iadesi expected: queue: billing ``` Autobench case defaults'i case ile merge eder. ### 5.4 Variant factors Variant, "hangi deney kosulu" sorusunun cevabidir: ```yaml variants: gpt_5_6_luna: label: OpenAI Luna model through OpenRouter factors: provider: value: openrouter semantic: llm.provider model: value: openrouter:openai/gpt-5.6-luna semantic: llm.model.name optimize: true temperature: value: 0.2 semantic: llm.temperature gemini_flash: label: Gemini flash model factors: provider: value: google semantic: llm.provider model: value: gemini-3-flash-preview semantic: llm.model.name optimize: true temperature: value: 0.2 semantic: llm.temperature ``` `optimize: true`, ileride autoptimize tarafinin "bu factor oynanabilir" diye okuyabilecegi bir sinyaldir. Autobench tek basina optimizasyon yapmaz; ama optimize edilebilir evidence toplar. --- ## 6. Python Task Runtime Task hedefleri `module:function` formatindadir: ```yaml run: python: app.benchmarks.support:run_ticket_case ``` Structured form: ```yaml task: kind: python target: app.benchmarks.support:run_ticket_case ``` Task sync olabilir: ```python def run_case(ctx, case): return {"ok": True} ``` Task async olabilir: ```python async def run_case(ctx, case): result = await my_async_app(case.input) return result ``` Task exception firlatirsa Autobench: - exception'i `ErrorRecord` olarak yakalar - run status'u errored yapar - onceki observation/artifact'lari korur - sonraki run'lara devam eder ### 6.1 Context'ten factor okuma ```python def run_case(ctx, case): model_name = ctx.factor("model") temperature = ctx.factor("temperature") return run_model(case.input, model=model_name, temperature=temperature) ``` Factor yoksa `KeyError` alirsin. Bu iyi bir sey; benchmark spec ve task arasindaki sozlesme bozuldu demektir. ### 6.2 Metric yazma ```python from autobench import Direction, ObservationRole, Semantic def run_case(ctx, case): output = do_work(case.input) ctx.metric( "answer_quality", output["score"], semantic_type=Semantic.QUALITY_SCORE, unit=None, direction=Direction.MAXIMIZE, role=ObservationRole.OBJECTIVE, ) return output ``` Metric isimleri lokal olabilir; semantic type kalici anlamdir. ### 6.3 Outcome yazma ```python def run_case(ctx, case): output = do_work(case.input) ctx.outcome(output["ok"]) return output ``` `ctx.outcome(True)` su anlama gelir: - metric name: `success` - semantic type: `result.success` - role: objective ### 6.4 Check yazma ```python from autobench import Semantic def run_case(ctx, case): output = do_work(case.input) expected = case.expected["label"] ctx.check( "label_matches", output["label"] == expected, reason=f"expected={expected}, got={output['label']}", semantic_type=Semantic.QUALITY_CORRECTNESS, ) return output ``` `check` constraint role ile metric yazar. Policy gibi daha genel gate'lerden farkli olarak task icinde domain-specific assertion kullanmak icin pratik bir helper'dir. ### 6.5 Artifact yazma ```python def run_case(ctx, case): output = do_work(case.input) ctx.artifact( "raw_response", output, media_type="application/x-yaml", tags={"kind": "debug"}, ) return output ``` Record sirasinda artifact payload'lari `artifacts/` altina materialize edilir. RunRecord sadece artifact reference tutar. ### 6.6 Span kullanma ```python from autobench import DurationMetricSpec, Semantic, SpanKind def run_case(ctx, case): with ctx.span( "retrieve_context", kind=SpanKind.RETRIEVER, input={"query": case.input["question"]}, duration_metric=DurationMetricSpec( name="retrieval_latency", semantic_type=Semantic.TIME_LATENCY, unit="s", ), ) as span: docs = retrieve(case.input["question"]) span.set_output({"doc_count": len(docs)}) span.metric("retrieved_docs", len(docs), semantic_type="retrieval.docs.count") with ctx.span("answer", kind=SpanKind.LLM) as span: answer = answer_question(case.input["question"], docs) span.set_output(answer) return {"answer": answer} ``` Span kaydi sunlari tasiyabilir: - id - name - kind: `agent`, `llm`, `tool`, `retriever`, `parser`, `workflow`, `custom` - parent id - started_at / ended_at - duration_seconds - input / output - attributes - usage - observations - artifacts - error - tags ### 6.7 Span duration framework'e ait Task icinde sunu yazmana gerek yok: ```python from time import perf_counter start = perf_counter() result = call() latency = perf_counter() - start ``` Onun yerine: ```python from autobench import DurationMetricSpec with ctx.span("call", duration_metric=DurationMetricSpec(name="call_latency")): result = call() ``` Span bitince `duration_seconds` dolacak ve ayrica metric observation yazilacaktir. --- ## 7. Measurement Helper Span tek bir operasyonun suresini olcer. Bazen benchmark'ta ayni callable'i birden fazla kez kosup median/p95/noise almak istersin. Autobench bunun icin generic measurement helper sunar. ```python from autobench import Semantic, measure_callable def run_case(ctx, case): def candidate(): return expensive_function(case.input["payload"]) measurement = measure_callable( candidate, warmup=3, repetitions=10, max_seconds=30.0, ) ctx.record_measurement( "candidate_runtime", measurement, semantic_type=Semantic.TIME_LATENCY, unit="ms", include_samples_artifact=True, ) return { "median_ms": measurement.median_ms, "p95_ms": measurement.p95_ms, "timed_out": measurement.timed_out, } ``` `Measurement` uzerindeki pratik property'ler: ```python measurement.samples_seconds measurement.samples_ms measurement.repetition_count measurement.median_seconds measurement.median_ms measurement.mean_ms measurement.min_ms measurement.max_ms measurement.p95_ms measurement.standard_deviation_ms measurement.range_noise_pct measurement.is_noisy(20.0) ``` Custom timer verebilirsin: ```python def my_timer(fn): started = monotonic_ns() fn() ended = monotonic_ns() return (ended - started) / 1_000_000_000 measurement = measure_callable(fn, repetitions=20, timer=my_timer) ``` Bu helper CUDA, browser, database veya LLM bilmez. Sadece callable olcer. Domain-specific setup/teardown task'in icinde kalir. --- ## 8. Observation Sistemi Observation Autobench'in en merkezi veri tiplerinden biridir: ```python from autobench import Observation, ObservationKind, Semantic obs = Observation( id="obs_1", name="input_tokens", kind=ObservationKind.METRIC, semantic_type=Semantic.LLM_TOKENS_INPUT, value=1234, unit="tokens", case_id="case_1", variant_id="gpt_5_6_luna", ) ``` Observation kind'lari: ```text metric factor event artifact ``` Observation role'lari: ```text objective constraint diagnostic ``` Observation source'lari: ```text task_observation score derived imported ``` Bu ayrim neden onemli? - Task metric'i ile scorer metric'i ayni sey degil. - Derived cost ile raw token observation ayni sey degil. - Imported trace observation'i ile local span metric'i ayni kaynak degil. - Objective optimize edilir, diagnostic sadece yorumlanir. Autobench reporting/projection katmani duplicate semantic metric'leri kaynak onceligine gore ele alir. Ornegin scorer output'u ayni semantic type icin task observation'ina gore daha guclu evidence olabilir. --- ## 9. Semantic Type Nedir? Autobench'te metric ismi ve semantic type farkli seylerdir. ```python ctx.metric("prompt_tokens", 1000, semantic_type="llm.tokens.input") ctx.metric("input_tokens", 1000, semantic_type="llm.tokens.input") ctx.metric("tokens_in", 1000, semantic_type="llm.tokens.input") ``` Bu uc metric farkli isimde olabilir ama ayni anlama gelir: ```text llm.tokens.input ``` Built-in semantic type ornekleri: ```python from autobench import Semantic Semantic.LLM_TOKENS_INPUT # "llm.tokens.input" Semantic.LLM_TOKENS_OUTPUT # "llm.tokens.output" Semantic.LLM_TOKENS_TOTAL # "llm.tokens.total" Semantic.LLM_MODEL_NAME # "llm.model.name" Semantic.LLM_PROVIDER # "llm.provider" Semantic.MONEY_COST # "money.cost" Semantic.TIME_LATENCY # "time.latency" Semantic.RESULT_SUCCESS # "result.success" Semantic.QUALITY_SCORE # "quality.score" Semantic.QUALITY_CORRECTNESS # "quality.correctness" Semantic.COVERAGE_RATIO # "coverage.ratio" Semantic.PROMPT_VERSION # "prompt.version" Semantic.DATASET_VERSION # "dataset.version" Semantic.AGENT_TASK_COMPLETION # "agent.task.completion" Semantic.AGENT_TOOL_ARGUMENT_CORRECTNESS ``` Semantic registry parent/alias bilir: ```python from autobench import DEFAULT_SEMANTIC_REGISTRY, Semantic registry = DEFAULT_SEMANTIC_REGISTRY assert registry.normalize("quality.answer") == Semantic.QUALITY_SCORE assert registry.is_a("agent.tool.argument.correctness", "quality.correctness") assert registry.is_a("optimization.cost", "money.cost") ``` Custom semantic type ekleyebilirsin: ```yaml benchmark: search-demo: semantic_registry: types: search.ndcg: parent: quality.score shape: number retrieval.docs.count: shape: integer aliases: ndcg_at_10: search.ndcg ``` Semantic awareness'in asil faydasi: - Farkli benchmark'larda metric isimleri degisse bile rapor ayni semantic type'a gore calisir. - Cost derivation dogru token/model/provider input'larini bulabilir. - Autoptimize ileride objective/constraint/diagnostic ayrimini anlayabilir. - Agentic metrics domain-specific isimlerden bagimsiz normalize edilebilir. --- ## 10. Scoring Task output'u tek basina benchmark sonucu degildir. Scoring katmani output'u, case expected degerlerini, span'lari ve custom scorer'lari kullanarak `ScoreRecord` uretir. ### 10.1 Output metric scorer YAML: ```yaml score: confidence: value: output.confidence semantic: quality.score role: diagnostic goal: maximize ``` Python: ```python from autobench import OutputMetricScorer, Semantic scorer = OutputMetricScorer( name="confidence", path="output.confidence", semantic_type=Semantic.QUALITY_SCORE, ) ``` `path` dotted path'tir. `output.confidence`, return edilen output dict veya object uzerinden cozulur. ### 10.2 Pass/fail scorer YAML: ```yaml score: success: pass: output.ok semantic: result.success ``` Python: ```python from autobench import PassFailScorer, Semantic PassFailScorer( name="success", path="output.ok", semantic_type=Semantic.RESULT_SUCCESS, ) ``` Bu scorer path'teki degeri `bool(...)` olarak yorumlar. ### 10.3 Exact scorer YAML: ```yaml score: answer_correct: exact: actual: output.answer expected: case.expected.answer semantic: quality.correctness ``` Python: ```python from autobench import ExactScorer, Semantic ExactScorer( name="answer_correct", actual="output.answer", expected="case.expected.answer", semantic_type=Semantic.QUALITY_CORRECTNESS, ) ``` Actual ve expected esit ise value `1.0`, degilse `0.0` olur. ### 10.4 Schema scorer YAML: ```yaml score: output_shape: schema: path: output schema: type: object required: [answer, citations] semantic: agent.output.structure.validity ``` Python: ```python from autobench import SchemaScorer, Semantic SchemaScorer( name="output_shape", path="output", schema={"type": "object", "required": ["answer", "citations"]}, semantic_type=Semantic.AGENT_OUTPUT_STRUCTURE_VALIDITY, ) ``` v0 yuzeyi object schema ve required key kontroluyle sinirlidir. ### 10.5 Python scorer YAML: ```yaml score: rubric_score: python: app.scorers:score_answer semantic: quality.score role: objective ``` `app/scorers.py`: ```python from autobench import ScoreRecord, Semantic def score_answer(call): output = call.output expected = call.case.expected score = 0.0 if expected["must_mention"] in output["answer"].lower(): score += 0.7 if output.get("citations"): score += 0.3 return ScoreRecord( name="rubric_score", semantic_type=Semantic.QUALITY_SCORE, value=score, actual_value=output, expected_value=expected, ) ``` Python scorer isterse direkt value donebilir: ```python def score_length(call): return min(len(call.output["answer"]) / 500, 1.0) ``` Async scorer da desteklenir: ```python async def score_with_judge(call): return await judge_answer(call.output, call.case.expected) ``` `ScoringCall` ile gelenler: ```python call.ctx call.task_result call.output call.case call.variant call.observations call.spans ``` ### 10.6 Span selector ile component-level scoring Bazen tum output'u degil belirli span'i score etmek istersin. ```yaml score: tool_arguments: expected_action: metric: arguments observed_kind: tool span: kind: tool name: lookup_user semantic: agent.tool.argument.correctness ``` `span` selector alanlari: ```yaml span: kind: tool name: lookup_user tag: phase: retrieval path: support_agent/lookup_user semantic_type: agent.tool.name ``` Selector, scorer'in sadece ilgili span'lari gormesini saglar. --- ## 11. Agentic Evidence Ve Expected Action Scoring Agent benchmark'larinda sadece final answer score etmek yetmez. Agent hangi tool'u cagirdi, argumanlari dogru muydu, siralama dogru muydu, bunlari da olcmek gerekir. ### 11.1 Tool span'i kaydetme ```python from autobench import SpanKind def run_case(ctx, case): with ctx.span("support_agent", kind=SpanKind.AGENT) as agent: with ctx.span( "lookup_user", kind=SpanKind.TOOL, input={"user_id": case.input["user_id"]}, tags={"tool": "lookup_user"}, ) as tool: result = lookup_user(case.input["user_id"]) tool.set_output(result) answer = build_answer(result) agent.set_output(answer) return {"answer": answer} ``` ### 11.2 Case expected action ```python from autobench import Case case = Case( id="refund_gold_user", input={"user_id": "u_123", "message": "I need a refund"}, expected={ "actions": [ { "id": "lookup_gold_user", "kind": "tool", "target": "lookup_user", "input": {"user_id": "u_123"}, "order": 1, "required": True, } ] }, ) ``` YAML: ```yaml cases: - id: refund_gold_user input: user_id: u_123 message: I need a refund expected: actions: - id: lookup_gold_user kind: tool target: lookup_user input: user_id: u_123 order: 1 required: true ``` ### 11.3 ExpectedActionScorer ```python from autobench import ExpectedActionScorer, Semantic, SpanSelector ExpectedActionScorer( name="tool_arguments", semantic_type=Semantic.AGENT_TOOL_ARGUMENT_CORRECTNESS, metric="arguments", observed_kind="tool", span=SpanSelector(kind="tool"), ) ``` YAML: ```yaml score: tool_selection: expected_action: metric: selection observed_kind: tool span: kind: tool semantic: agent.tool.selection.correctness tool_arguments: expected_action: metric: arguments observed_kind: tool span: kind: tool semantic: agent.tool.argument.correctness tool_sequence: expected_action: metric: sequence observed_kind: tool span: kind: tool semantic: agent.tool.sequence.correctness ``` Metric secenekleri: ```text selection arguments sequence ``` Bu yuzey DeepEval tarzi agent eval ihtiyaclarina benzese de Autobench bunu generic semantic evidence modeline koyar. Tool correctness sadece LLM degil, herhangi bir workflow tool sistemi icin de kullanilabilir. --- ## 12. Token Cost Derivation Autobench cost'u kendisi "bilir" gibi davranmaz. Cost, semantic input'lardan turetilen bir metriktir: Gerekli semantic input'lar: ```text llm.tokens.input llm.tokens.output llm.provider llm.model.name ``` Task bu degerleri observation olarak yazar: ```python from autobench import Semantic def run_case(ctx, case): result = call_llm(case.input["prompt"]) usage = result.usage ctx.metric("input_tokens", usage.input_tokens, semantic_type=Semantic.LLM_TOKENS_INPUT) ctx.metric("output_tokens", usage.output_tokens, semantic_type=Semantic.LLM_TOKENS_OUTPUT) ctx.factor_observation("provider", "openrouter", semantic_type=Semantic.LLM_PROVIDER) ctx.factor_observation("model", "openrouter:openai/gpt-5.6-luna", semantic_type=Semantic.LLM_MODEL_NAME) return {"answer": result.output} ``` Pricing YAML: ```yaml pricing: provider: openrouter source: manual models: openai/gpt-5.6-luna: input: unit: mtok price: 0.4 output: unit: mtok price: 1.6 ``` Tiered pricing: ```yaml pricing: models: openai/gpt-tiered: input: unit: mtok price: 1.0 output: unit: mtok tiers: - up_to: 500 price: 4.0 - price: 2.0 cache_read: unit: mtok price: 0.2 cache_write: unit: mtok price: 0.5 ``` Benchmark spec: ```yaml benchmark: llm-cost-demo: cases: - id: case_1 input: prompt: Say hello run: python: app.tasks:run_case variants: default: {} derive: - kind: token_cost pricing: pricing/models.yaml output: name: cost semantic_type: money.cost unit: usd inputs: input_tokens: llm.tokens.input output_tokens: llm.tokens.output provider: llm.provider model: llm.model.name ``` Minimal formda `output` ve `inputs` verilmezse default kullanilir: ```yaml derive: - kind: token_cost pricing: pricing/models.yaml ``` Cost bulunamazsa Autobench uydurma cost yazmaz. Diagnostic observation uretir: - `token_cost_missing_inputs` - `token_cost_unknown_pricing` - `token_cost_missing_rates` Bu tasarim onemli: Benchmark "bilmedigi seyi" tahmin etmez; eksik evidence'i gorunur yapar. --- ## 13. Paired Baseline Post-Derivation Normal deriver tek run icindeki observation'lardan metric uretir. Paired baseline ise experiment bittikten sonra run'lari birbirine bakarak karsilastirir. Klasik use case: - baseline latency: 100 ms - candidate latency: 50 ms - speedup: 2.0 Task: ```python from autobench import Semantic def run_case(ctx, case): if ctx.variant.id == "baseline": latency_ms = case.input["baseline_ms"] else: latency_ms = case.input["candidate_ms"] ctx.metric( "median_latency", latency_ms, semantic_type=Semantic.TIME_LATENCY, unit="ms", ) return {"latency_ms": latency_ms} ``` YAML: ```yaml benchmark: latency-comparison: cases: - id: easy input: baseline_ms: 100 candidate_ms: 50 - id: hard input: baseline_ms: 240 candidate_ms: 180 run: python: app.tasks:run_case variants: baseline: {} candidate: {} post_derive: - kind: paired_baseline baseline_variant: baseline match_on: - kind: case_id metric: time.latency output: name: speedup semantic_type: performance.speedup formula: baseline_over_candidate report: leaderboard: show: avg_speedup: metric: performance.speedup aggregate: mean matrix: metric: performance.speedup ``` Formula secenekleri: ```text baseline_over_candidate candidate_over_baseline candidate_minus_baseline baseline_minus_candidate percent_change_from_baseline ``` Eslestirme sadece case id olmak zorunda degil. Factor da kullanilabilir: ```yaml match_on: - kind: case_id - kind: factor name: workload.size ``` Threshold ve verdict: ```yaml post_derive: - kind: paired_baseline baseline_variant: baseline match_on: - kind: case_id metric: time.latency output: name: speedup semantic_type: performance.speedup formula: baseline_over_candidate threshold: kind: relative_noise pct: 2.0 verdict: output: name: latency_verdict semantic_type: comparison.verdict threshold: kind: relative_noise pct: 2.0 ``` Missing behavior: ```yaml missing: diagnostic zero_division: diagnostic diagnostics_name: paired_baseline_unavailable ``` veya: ```yaml missing: skip zero_division: skip ``` Autobench burada causal attribution iddia etmez. Sadece evidence uretir: > Baseline ve candidate ayni case/factor match uzerinde karsilastirildi, > speedup su kadar. "Bu degisime kesin olarak hangi factor sebep oldu?" sorusu autoptimize veya daha kontrollu deney tasarimi konusudur. --- ## 14. Policy Checks Policy, release gate gibi dusunulebilir. Bir semantic metric belirli bir sarti saglamali. Python: ```python from autobench import PolicySpec, Semantic policies = [ PolicySpec( name="minimum_quality", metric=Semantic.QUALITY_SCORE, must_greater_equal=0.8, ), PolicySpec( name="maximum_cost", metric=Semantic.MONEY_COST, must_less_equal=0.002, ), ] ``` YAML: ```yaml policies: - name: minimum_quality metric: quality.score must_greater_equal: 0.8 - name: maximum_cost metric: money.cost must_less_equal: 0.002 - name: queue_allowed metric: support.queue must_in: [billing, account, general] ``` Desteklenen requirement alanlari: ```text must_equal must_not_equal must_greater must_greater_equal must_less must_less_equal must_in must_not_in must_between ``` `must_between`: ```yaml policies: - name: confidence_band metric: quality.score must_between: min: 0.7 max: 1.0 inclusive: true ``` Policy sonucu `policy.result` semantic type'iyle derived observation olarak eklenir. Policy fail, final run status'u etkileyebilir. --- ## 15. Reporting Report spec su an dort ana gorunum bilir: - leaderboard - case matrix - comparison - distribution ### 15.1 Leaderboard ```yaml report: leaderboard: show: pass_rate: metric: result.success aggregate: ratio_true avg_quality: metric: quality.score aggregate: mean total_cost: metric: money.cost aggregate: sum ``` Python model: ```python from autobench import LeaderboardReportSpec, MetricAggregation, ReportSpec, Semantic report_spec = ReportSpec( leaderboard=LeaderboardReportSpec( metrics=( MetricAggregation( name="pass_rate", semantic_type=Semantic.RESULT_SUCCESS, fn="ratio_true", ), MetricAggregation( name="avg_quality", semantic_type=Semantic.QUALITY_SCORE, fn="mean", ), ) ) ) ``` Aggregation fonksiyonlari: ```text count mean sum min max median p95 stddev geomean ratio_true ``` ### 15.2 Case matrix ```yaml report: matrix: metric: quality.correctness ``` Bu case x variant matrisi uretir. Bir case hangi variant'ta iyi/kotu gitti gorulebilir. ### 15.3 Comparison ```yaml report: compare: baseline -> candidate: show: avg_speedup: metric: performance.speedup aggregate: mean avg_quality: metric: quality.score aggregate: mean ``` CLI: ```bash uv run autobench compare runs/latency --baseline baseline --candidate candidate ``` Compare, factor delta ve metric delta gosterir ama causal claim yapmaz. ### 15.4 Distribution Structured model: ```python from autobench import DistributionReportSpec, ReportSpec, Semantic ReportSpec( distributions=( DistributionReportSpec( name="latency_distribution", semantic_type=Semantic.TIME_LATENCY, summaries=("min", "median", "p95", "max"), ), ) ) ``` YAML structured form: ```yaml reports: distributions: - name: latency_distribution semantic_type: time.latency summaries: [min, median, p95, max] ``` ### 15.5 Export ```bash uv run autobench export runs/support-routing --format markdown --path report.md uv run autobench export runs/support-routing --format yaml --path report.yaml uv run autobench export runs/support-routing --format csv --path runs.csv ``` Export stdout'a raw YAML/Markdown basmaz. Dosyaya yazar, terminalde Rich preview gosterir. --- ## 16. Recording Ve Replay `autobench run --record runs/demo` su dosyalari yazar: ```text runs/demo/ experiment.yaml summary.yaml cases/ / / run.yaml artifacts/ ... ``` RunRecord alanlari: ```text record_version run_id experiment_id benchmark_id case_id variant_id status evaluation_status task_status case task_output observations scores spans artifacts factors asset_versions parent_run_id errors error ``` ExperimentRecord alanlari: ```text record_version experiment_id benchmark_id plan environment semantic_registry report_spec_data spec_snapshot spec_hash file_hashes run_paths run_count passed_count failed_count errored_count skipped_count ``` Replay: ```bash uv run autobench replay runs/demo ``` Replay task module'unu import etmez. Bu cok kritik: - Model API key gerekmez. - External service tekrar cagirilmaz. - Onceki experiment reproducible raporlanir. - Eski app kodu degismis olsa bile record okunabilir. Programmatic replay: ```python from pathlib import Path from autobench import build_report, load_experiment_record, replay_experiment record = load_experiment_record(Path("runs/demo")) result = replay_experiment(record, root_dir=Path("runs/demo")) report = build_report(result) ``` Programmatic record: ```python from pathlib import Path from autobench import collect_benchmark_source_files, record_experiment, run_benchmark_path spec_path = Path("autobench.yaml") result = run_benchmark_path(spec_path, concurrency_limit=2) source_files = collect_benchmark_source_files(spec_path) record = record_experiment(result, Path("runs/demo"), source_files=source_files) ``` Record append-only'dir. Aynı output dir'de `experiment.yaml` varsa `RecordingError` alirsin. Bu, eski evidence'in sessizce overwrite edilmesini engeller. --- ## 17. Python Builder API YAML ana kaynak olmali, ama Python builder da var. Basit benchmark'lar icin ergonomiktir. ```python from autobench import Benchmark, Case, ExactScorer, FactorValue, PassFailScorer, Semantic, Variant result = ( Benchmark("builder-demo") .description("Small builder API demo.") .dataset( [ Case( id="case_1", input={"text": "refund please"}, expected={"queue": "billing"}, ) ] ) .variants( [ Variant( id="route_v1", factors=[ FactorValue( name="prompt_version", value="route-v1", semantic_type=Semantic.PROMPT_VERSION, ) ], ), { "id": "route_v2", "factors": { "prompt_version": { "value": "route-v2", "semantic_type": Semantic.PROMPT_VERSION, } }, }, ] ) .task("app.benchmarks.support:run_ticket_case") .scoring( [ PassFailScorer( name="success", path="output.matched", semantic_type=Semantic.RESULT_SUCCESS, ), ExactScorer( name="queue_correct", actual="output.queue", expected="case.expected.queue", semantic_type=Semantic.QUALITY_CORRECTNESS, ), ] ) .run(concurrency_limit=1) ) ``` Builder su anda temel spec parcalarini kurar: - benchmark id/description - dataset - variants - task - scoring - derive - run/run_async Report/post-derive/policy gibi daha gelismis yuzeylerde YAML veya Pydantic model'leri dogrudan kullanmak daha nettir. --- ## 18. Programmatic Full Spec Her seyi Python model'leriyle de kurabilirsin: ```python from pathlib import Path from autobench import ( BenchmarkInfo, BenchmarkSpec, Case, ComparisonReportSpec, DatasetSpec, DerivedMetricOutput, Direction, FactorValue, LeaderboardReportSpec, MetricAggregation, ObservationRole, PairedBaselineDeriverSpec, PassFailScorer, PolicySpec, ReportSpec, RunMatchKey, Semantic, TaskSpec, TokenCostDeriverSpec, TokenCostInputs, Variant, run_benchmark_spec, ) spec = BenchmarkSpec( benchmark=BenchmarkInfo( id="programmatic-demo", description="Fully typed benchmark spec.", ), dataset=DatasetSpec( cases=[ Case(id="case_1", input={"prompt": "Say hi"}, expected={"ok": True}), ] ), task=TaskSpec(kind="python", target="app.tasks:run_case"), variants=[ Variant( id="baseline", factors=[ FactorValue( name="model", value="gpt-baseline", semantic_type=Semantic.LLM_MODEL_NAME, ) ], ), Variant( id="candidate", factors=[ FactorValue( name="model", value="gpt-candidate", semantic_type=Semantic.LLM_MODEL_NAME, ) ], ), ], scoring=[ PassFailScorer( name="success", path="output.ok", semantic_type=Semantic.RESULT_SUCCESS, role=ObservationRole.OBJECTIVE, direction=Direction.MAXIMIZE, ) ], derive=[ TokenCostDeriverSpec( pricing="pricing/models.yaml", output=DerivedMetricOutput( name="cost", semantic_type=Semantic.MONEY_COST, unit="usd", ), inputs=TokenCostInputs(), ) ], post_derive=[ PairedBaselineDeriverSpec( baseline_variant="baseline", match_on=(RunMatchKey(kind="case_id"),), metric=Semantic.TIME_LATENCY, output=DerivedMetricOutput( name="speedup", semantic_type="performance.speedup", ), ) ], policies=[ PolicySpec( name="must_succeed", metric=Semantic.RESULT_SUCCESS, must_equal=True, ) ], reports=ReportSpec( leaderboard=LeaderboardReportSpec( metrics=( MetricAggregation( name="success_rate", semantic_type=Semantic.RESULT_SUCCESS, fn="ratio_true", ), MetricAggregation( name="total_cost", semantic_type=Semantic.MONEY_COST, fn="sum", ), ) ), comparisons=( ComparisonReportSpec( baseline="baseline", candidate="candidate", metrics=( MetricAggregation( name="avg_speedup", semantic_type="performance.speedup", fn="mean", ), ), ), ), ), ) ``` Async run: ```python result = await run_benchmark_spec(spec, experiment_id="exp_programmatic") ``` Sync wrapper kullanmak istersen: ```python import asyncio result = asyncio.run(run_benchmark_spec(spec)) ``` --- ## 19. Asset Tracking Autobench'in en onemli uzun vadeli parcalarindan biri asset tracking'dir. Prompt, tool, output type, dataclass ve typed class gibi uygulama varliklarini version'layabilir. ### 19.1 Prompt tracking Text inline: ```python from autobench import track SYSTEM_PROMPT = track.prompt( name="support_router_system", text="Route the support ticket to billing, account, or general.", ) print(str(SYSTEM_PROMPT)) print(SYSTEM_PROMPT.raw) print(SYSTEM_PROMPT.version) ``` Dosyadan: ```python from autobench import track SYSTEM_PROMPT = track.prompt( name="support_router_system", source="./prompts/support_router.md", ) ``` `raw` property, prompt text'ine dogrudan ulasmak icindir. `str(prompt)` da ayni metni verir. ### 19.2 Tool tracking ```python from typing import Literal from autobench import track Queue = Literal["billing", "account", "general"] @track.tool def assign_queue(ticket_id: str, queue: Queue, priority: int = 0) -> dict[str, object]: """Assign a support ticket to a queue.""" return {"ticket_id": ticket_id, "queue": queue, "priority": priority} ``` Autobench tool icin sunlari cikarmaya calisir: - name - qualname - docstring - param schema - param annotation - literal choices - required/default bilgisi - return annotation - return type asset id - source hash - content hash ### 19.3 Pydantic model tracking ```python from typing import Literal from autobench import track from pydantic import BaseModel, Field Queue = Literal["billing", "account", "general"] @track.type class RoutingOutput(BaseModel): queue: Queue = Field(description="Queue selected for the ticket.") confidence: float = Field(ge=0, le=1, description="Router confidence.") reasons: list[str] = Field(default_factory=list) ``` Pydantic model icin hash JSON schema / structured fields uzerinden uretilir. Field asset'leri sunlari tutabilir: - field name - annotation - required/default/default_factory - description - examples - alias - constraints - literal choices ### 19.4 Dataclass tracking Iki sekilde kullanabilirsin. Standart dataclass ustune: ```python from dataclasses import dataclass from autobench import track @track.type @dataclass(frozen=True) class Ticket: id: str subject: str ``` Daha iyi DX icin Autobench dataclass decorator'i: ```python from autobench import track @track.dataclass(frozen=True, slots=True) class Ticket: id: str subject: str ``` Bu hem dataclass'i uygular hem de type asset olarak track eder. Decorator metadata'si de asset metadata'ya girer. ### 19.5 Generic class decorator Kendi class decorator'ini kullanmak istersen: ```python from attrs import define from autobench import track @track.decorate_type(define, frozen=True) class AttrsTicket: id: str subject: str ``` Bu pattern su durumlarda ise yarar: - `dataclass` - `attrs.define` - custom class transformer - framework-specific typed class decorator'leri ### 19.6 Asset version'lari run'a baglama ```python from autobench import track ROUTER_PROMPT = track.prompt(name="router", source="./prompts/router.md") @track.tool def lookup_user(user_id: str) -> dict[str, str]: return {"tier": "gold"} def run_case(ctx, case): ctx.attach_tracked_asset(ROUTER_PROMPT) ctx.attach_tracked_asset(lookup_user) return {"ok": True} ``` RunRecord icinde `asset_versions` alanina bu version'lar girer. Boylece daha sonra "bu run hangi prompt/tool/type version'i ile kosuldu?" sorusu cevaplanir. ### 19.7 Asset registry'yi diske yazma ```python from pathlib import Path from autobench import track track.write_assets(Path(".autobench/assets")) ``` Yazilan yapinin amaci: - `index.yaml`: asset listesi ve latest version'lar - `.yaml`: asset metadata, versions, diffs Autobench version hash ve source hash tutar. Prompt gibi text assetlerde raw icerik metadata'da tutulur. Type assetlerde structured field/schema bilgisi hash'e girer. --- ## 20. Instrumentation Her metric'i task icinde elle yazmak istemeyebilirsin. Autobench runtime instrumentation ile class method'larindan metric/factor toplayabilir. Basit servis: ```python class LLMClient: def complete(self, prompt: str): return { "text": "hello", "usage": { "input_tokens": 10, "output_tokens": 5, "model": "gpt-demo", "provider": "openai", }, } ``` Instrumentation: ```python from autobench import InstrumentFactorSpec, InstrumentMetricSpec, Semantic, instrument_method handle = instrument_method( LLMClient, "complete", metrics=[ InstrumentMetricSpec( name="input_tokens", semantic_type=Semantic.LLM_TOKENS_INPUT, value_path="result.usage.input_tokens", ), InstrumentMetricSpec( name="output_tokens", semantic_type=Semantic.LLM_TOKENS_OUTPUT, value_factory=lambda call: call.result["usage"]["output_tokens"], ), ], factors=[ InstrumentFactorSpec( name="model", semantic_type=Semantic.LLM_MODEL_NAME, value_path="result.usage.model", ), InstrumentFactorSpec( name="provider", semantic_type=Semantic.LLM_PROVIDER, value_path="result.usage.provider", ), ], ) ``` Task icinde: ```python def run_case(ctx, case): client = LLMClient() return client.complete(case.input["prompt"]) ``` Autobench pipeline calisirken active `RunContext` contextvar olarak set edilir. Instrumented method cagrildiginda metric/factor observation'lari aktif ctx'e yazilir. `value_factory` Python API'deki typed callback seam'idir. `value_path` ise mapping key, attribute ve sifir argumanli accessor zincirlerini declarative olarak cozer. YAML tarafinda arbitrary expression calistirilmaz. Method wrapper instance/static/class/inherited method'lari, sync/async call'lari, generator ve async generator stream'lerini ve sync/async context manager'lari destekler. Stream span'i iterator olusturulunca degil gercekten bittiginde, hata verdiginde veya erken kapandiginda kapanir. Handle kapatma: ```python handle.close() ``` Context manager: ```python with instrument_method(LLMClient, "complete", metrics=[...]): result = run_benchmark_path(Path("autobench.yaml")) ``` Not: Bu OTel degildir. Autobench'in kendi lightweight instrumentation yuzeyidir. OTel/Logfire bridge future plan olarak dusunulur. Reusable entegrasyonlar `InstrumentorInfo`, `Compatibility`, `Instrumentor`, `InstrumentationRuntime` ve `InstrumentationManager` kontratini kullanir. Manager package version'ini hook/patch kurmadan once denetler, duplicate install'lari reference count ile birlestirir ve son handle kapandiginda native callback'i veya exact descriptor'u restore eder. Task-local `suppress_instrumentation("family")` sadece eslesen instrumentor/family'yi gecici olarak susturur. ### 20.1 Native instrumentor DSL ve doctor Pydantic AI, OpenAI client, OpenAI Agents ve HTTPX instrumentor'lari typed Python ayarlariyla veya benchmark YAML icinden secilebilir: ```yaml instrumentation: all: exclude: [httpx] strict: false pydantic_ai: {} openai: {} httpx: capture: path: hash response_headers: [x-request-id] ``` Python builder'da ortamdaki tum kurulu ve uyumlu built-in entegrasyonlar `Benchmark.instrument_all()` ile acilir. `exclude` discovery alanini daraltir; `strict=True` kurulamayan ilk entegrasyonda hata verir. Default mod kurulamayan entegrasyonu atlar ve nedenini her run'a diagnostic evidence olarak yazar. Explicit config, `false` dahil, discovery sonucunu override eder; ayni ID'ye sahip custom runtime instrumentor da otomatik kurulumu engeller. Tekil config `Benchmark.instrument(...)` ile verilir. Ayarlar `BenchmarkSpec` icinde serialize edilir; custom `Instrumentor` instance'lari ayni method'dan gecse de runtime-only kalir. Pipeline instrumentor'lari tum matrix'ten once kurar ve hata halinde de kapatir. `autobench instrumentation doctor` kurulu/eksik extra'lari, version uyumlulugunu, layer/mechanism'i, sync/async/streaming capability'lerini, semantic aileleri ve capture default'larini Rich tablolarla gosterir. Eksik SDK import edilmez. `autobench instrumentation trace RUN_DIR` ise provider SDK ve task import etmeden recorded ABP trace composition ve partial state'i gosterir. HTTPX default'u body ve header toplamaz; path hash'lenir. Secret header/body alanlari explicit capture'da bile redact edilir ve body capture bounded'dir. Pydantic AI -> OpenAI -> HTTPX layer'lari parent-child olarak compose edilir; transport token/cost uretmez ve aggregate/direct accounting double count'i engeller. Detayli kontrat: [Native Instrumentation](native-instrumentation.md). ### 20.2 ABP trace extraction ve accounting Instrumentor ham ABP fact'lerini toplar; extractor tamamlanmis immutable trace'ten tekrar kullanilabilir semantic evidence uretir: ```python from autobench import CompositeExtractor, SignalExtractor, SpanExtractor, UsageExtractor extractor = CompositeExtractor( SignalExtractor(), SpanExtractor(), UsageExtractor(), ) ``` - `SignalExtractor`: measurement/event observation'larini scope, layer, instrumentor ve logical operation provenance'i ile geri kurar. - `SpanExtractor`: direct span latency, operation count, max depth/fan-out, critical path, parallelism, incomplete work, retry/recovery, validation, approval, tool ve reference metric'lerini uretir. - `UsageExtractor`: LLM request/token accounting'i ile requested model, response model ve provider factor'lerini uretir. Cost uretmez; cost mevcut pricing deriver'inin sorumlulugudur. Parent aggregate ile child direct usage toplanmaz. Her semantic icin tek abstraction boundary secilir; logical operation ID'si ayni olan direct olcumler deduplicate edilir. Esdeger direct degerler cakisiyor ve unique bir authority yoksa total uydurulmaz, `ambiguous_direct_measurement` diagnostic'i uretilir. Aggregate deger direct toplamla uyusmazsa `aggregate_measurement_mismatch` kaydedilir. Extractor isim ve version'a sahiptir. `replay_extraction()` yeni bir derived RunRecord yazar; ayni extractor'in yeni version'i onceki observation'lari yeni derived record'da degistirir ve parent lineage'i korur. --- ## 21. Pydantic AI Usage Bridge Pydantic AI result usage bilgisini Autobench semantic observation'a cevirmek icin helper vardir: ```python from autobench import PydanticAIUsage, record_pydantic_ai_usage def run_case(ctx, case): result = agent.run_sync(case.input["prompt"]) usage = result.usage() record_pydantic_ai_usage( ctx, PydanticAIUsage( requests=usage.requests, input_tokens=usage.input_tokens, output_tokens=usage.output_tokens, total_tokens=usage.total_tokens, model_name="google/gemini-3-flash-preview", provider="openrouter", ), ) return {"answer": result.output} ``` Eger span icinde kullaniyorsan: ```python with ctx.span("agent_run", kind="agent") as span: result = agent.run_sync(case.input["prompt"]) record_pydantic_ai_usage(ctx, usage, span_id=span.id) ``` Bu helper su semantic type'lari yazar: - `llm.tokens.input` - `llm.tokens.output` - `llm.tokens.total` - `llm.model.name` - `llm.provider` --- ## 22. TraceEnvelope Dis frameworklerden trace getirmek icin minimal bir envelope var. ```python from datetime import UTC, datetime from autobench import SpanRecord, TraceEnvelope, attach_trace trace = TraceEnvelope( trace_id="trace_123", name="external_agent_run", spans=( SpanRecord( id="span_llm_1", name="llm_call", kind="llm", started_at=datetime.now(UTC), duration_seconds=0.42, usage={"input_tokens": 200, "output_tokens": 40}, attributes={"model": "gpt-demo", "provider": "openai"}, ), ), ) def run_case(ctx, case): attach_trace(ctx, trace) return {"ok": True} ``` `trace_to_observations` LLM span usage'larini semantic observation'a cevirir: - `prompt_tokens` veya `input_tokens` -> `llm.tokens.input` - `completion_tokens` veya `output_tokens` -> `llm.tokens.output` - `total_tokens` -> `llm.tokens.total` - `model` veya `model_name` -> `llm.model.name` - `provider` -> `llm.provider` - span duration -> `time.latency` Bu bir full tracing platformu degil; recorded evidence'a trace eklemek icin hafif bir adapter yuzeyidir. --- ## 23. Pydantic Evals Payload Bridge Autobench Pydantic Evals'i ana runtime olarak zorlamaz. Ama payload adapter vardir. ```python from autobench import PydanticEvalsBridge, load_benchmark_spec spec = load_benchmark_spec("autobench.yaml") bridge = PydanticEvalsBridge() if bridge.is_available(): dataset_payload = bridge.dataset_payload(spec) print(dataset_payload.name) print(dataset_payload.cases[0].inputs) ``` Case payload: ```python from autobench import Case, PydanticEvalsBridge bridge = PydanticEvalsBridge() payload = bridge.case_payload( Case( id="case_1", input={"question": "2+2?"}, expected={"answer": "4"}, tags=["math"], ) ) ``` Payload alanlari: ```text name inputs expected_output metadata ``` Pydantic Evals yuklu degilse: ```python bridge.require_module() ``` `PydanticEvalsUnavailableError` firlatir. --- ## 24. Production Sample Ve Synthetic Case Helper'lari Gercek production data'dan benchmark case uretmek isteyebilirsin. ```python from datetime import datetime, UTC from autobench import ProductionSample, SampleReason, ReviewStatus, sample_to_case sample = ProductionSample( id="prod_001", input={"message": "refund please"}, output={"queue": "billing"}, expected={"queue": "billing"}, reason=SampleReason.FAILURE_ONLY, review_status=ReviewStatus.CANDIDATE, timestamp=datetime.now(UTC), privacy_tags=("pii_redacted",), ) case = sample_to_case(sample) ``` `case.metadata` icine su bilgiler girer: - `source: production` - `sample_reason` - `review_status` - `timestamp` - `privacy_tags` - `trace_id` varsa Bir batch'i policy ile filtrelemek: ```python from autobench import SamplingPolicy, SampleReason, samples_to_cases cases = samples_to_cases( samples, policy=SamplingPolicy( reasons=(SampleReason.FAILURE_ONLY, SampleReason.HIGH_COST), max_samples=100, ), ) ``` Synthetic/generated case isaretleme: ```python from autobench import Case, generated_batch_from_cases, mark_generated_case case = Case(id="synthetic_1", input={"question": "..."}, expected={"answer": "..."}) marked = mark_generated_case( case, generator_asset_version="prompt.generator@abc123", model_provider="openrouter", model_name="openai/gpt-5.6-luna", ) batch = generated_batch_from_cases( [case], generator_asset_version="prompt.generator@abc123", model_provider="openrouter", model_name="openai/gpt-5.6-luna", ) ``` Bu helper'lar data lineage icindir. Autobench synthetic data generator calistirmaz; generated case'i evidence modelinde dogru isaretler. --- ## 25. Metric Packs Metric pack, belli bir domain icin semantic registry delta, scorer factory isimleri, default report metric'leri ve feedback extractor isimlerini paketler. Built-in pack'ler: ```python from autobench import DEFAULT_METRIC_PACKS print(DEFAULT_METRIC_PACKS.names()) ``` Beklenen pack id'leri: ```text agentic structured_output llm_usage performance ``` Pack okuma: ```python from autobench import builtin_metric_pack_registry registry = builtin_metric_pack_registry() agentic = registry.require("agentic") print(agentic.default_report_metrics) print(agentic.scorer_factories) ``` Semantic registry merge: ```python semantic_registry = registry.semantic_registry_for(["agentic", "llm_usage"]) ``` Bu henuz full plugin sistemi degil. Ama ileride `autobench-ai`, `autobench-agentic`, `autobench-performance` gibi domain paketlerinin tasiyabilecegi sozlesmenin cekirdegi. --- ## 26. Pricing Source Helper'lari Autobench pricing library olmak istemez. Ama LLM fiyat tablolarini tek formata normalize etmeye yardim eden helper'lar vardir. Manual load/dump: ```python from pathlib import Path from autobench import load_pricing_table, dump_pricing_table table = load_pricing_table(Path("pricing/models.yaml")) dump_pricing_table(table, Path("pricing/normalized.yaml")) ``` Static source: ```python from autobench import StaticPriceSource source = StaticPriceSource(Path("pricing/models.yaml")) table = source.load() ``` LLMPrices / GenAIPrices source adapter'lari: ```python from autobench import GenAIPricesSource, LLMPricesSource genai_table = GenAIPricesSource(Path("genai-prices.json")).load() llm_prices_table = LLMPricesSource(Path("llm-prices.json")).load() ``` Model id normalizasyonu: ```text provider:model provider/model model with provider field aliases ``` Fikir su: kullanici isterse fiyatlari elle yazar, isterse source adapter ile normalize eder. Autobench kendi basina pricing ownership almaz. --- ## 27. Feedback Records Autobench recorded run'dan optimization icin feedback payload'u da uretmeye baslar. ```python from pathlib import Path from autobench import build_optimization_feedback_input, load_run_record record = load_run_record(Path("runs/demo/cases/case_1/variant_1/run.yaml")) feedback = build_optimization_feedback_input(record) ``` `FeedbackRecord` alanlari: ```text case_id variant_id score_name semantic_type score passed failure_category reason span ``` Basarili case icin `failure_category` `None` olur. Error varsa `error`, assertion fail varsa `assertion_failure`, low score varsa `low_score` gibi kategoriler kullanilir. Bu parca ileride autoptimize icin onemli olacak. Ama v0.1'de sadece evidence'dan feedback input uretme seviyesindedir. --- ## 28. Run Status Model Autobench farkli status katmanlarini ayirir. Task status: ```text passed failed errored skipped ``` Evaluation status: ```text passed failed errored skipped ``` Run status: ```text passed failed errored skipped ``` Neden ayri? - Task tamamlanabilir ama scorer fail olabilir. - Task tamamlanabilir ama policy fail olabilir. - Task hic tanimli degilse skipped olabilir. - Exception varsa errored olabilir. Bu ayrim RunRecord'da gorunur: ```yaml run: status: failed outcome: evaluation: failed task: passed ``` Bu ornekte uygulama calismis, ama evaluation basarisiz olmustur. --- ## 29. YAML Record Ornegi Bir run record kabaca soyle gorunur: ```yaml record: type: run version: 4 protocol: name: abp version: 1 semantic_registry: 1 run: id: run_refund_request_route_v2 experiment: exp_support_routing_20260512T120000Z benchmark: support-routing case: refund_request variant: route_v2 status: passed outcome: evaluation: passed task: passed case: id: refund_request input: subject: Duplicate card charge body: I was billed twice yesterday and need a refund. expected: queue: billing tags: - billing - refund variant: id: route_v2 label: improved routing prompt factors: prompt_version: value: route-v2 semantic: prompt.version optimize: true scores: matched: value: true semantic: result.success role: objective queue_correct: value: 1.0 semantic: quality.correctness actual: billing expected: billing metrics: objectives: matched: value: true semantic: result.success queue_correct: value: 1.0 semantic: quality.correctness diagnostics: confidence: value: 0.94 semantic: quality.score routing_duration: value: 0.0021 semantic: time.latency unit: s spans: route_ticket: kind: workflow input: subject: Duplicate card charge body: I was billed twice yesterday and need a refund. output: queue: billing confidence: 0.94 matched: true duration: 0.0021 output: queue: billing confidence: 0.94 matched: true ``` Gercek dosyadaki alanlar daha ayrintili olabilir; ama DSL-like hedef budur: insan okuyabilsin, makine de deserialize edebilsin. --- ## 30. End-to-End Ornek: LLM Reply Quality Benchmark Bu ornek bir LLM reply generator'in kalite, maliyet, latency ve policy kontrollerini ayni benchmark'ta toplar. ### 30.1 Task `app/reply.py`: ```python from __future__ import annotations from dataclasses import dataclass @dataclass class FakeUsage: input_tokens: int output_tokens: int total_tokens: int @dataclass class FakeResult: output: str usage: FakeUsage def generate_reply(prompt: str, *, model: str, tone: str) -> FakeResult: if tone == "concise": output = "We can help with that. Please share your order id." else: output = ( "Thanks for reaching out. We can help with that refund request. " "Please share your order id so our billing team can review it." ) return FakeResult( output=output, usage=FakeUsage( input_tokens=len(prompt.split()) + 20, output_tokens=len(output.split()), total_tokens=len(prompt.split()) + len(output.split()) + 20, ), ) ``` `app/benchmarks/reply.py`: ```python from __future__ import annotations from autobench import DurationMetricSpec, PydanticAIUsage, Semantic, SpanKind, record_pydantic_ai_usage from app.reply import generate_reply def run_reply_case(ctx, case): model = ctx.factor("model") tone = ctx.factor("tone") with ctx.span( "reply_generation", kind=SpanKind.LLM, input=case.input, attributes={"model": model, "provider": "demo"}, duration_metric=DurationMetricSpec(name="reply_latency"), ) as span: prompt = case.input["message"] result = generate_reply(prompt, model=model, tone=tone) span.set_output(result.output) span.set_usage("input_tokens", result.usage.input_tokens) span.set_usage("output_tokens", result.usage.output_tokens) record_pydantic_ai_usage( ctx, PydanticAIUsage( requests=1, input_tokens=result.usage.input_tokens, output_tokens=result.usage.output_tokens, total_tokens=result.usage.total_tokens, model_name=model, provider="demo", ), span_id=span.id, ) contains_required_phrase = case.expected["required_phrase"] in result.output.lower() ctx.check( "required_phrase_present", contains_required_phrase, semantic_type=Semantic.QUALITY_CORRECTNESS, ) return { "reply": result.output, "contains_required_phrase": contains_required_phrase, } ``` ### 30.2 Pricing `pricing/demo-models.yaml`: ```yaml pricing: provider: demo source: manual models: demo/small: input: unit: mtok price: 0.1 output: unit: mtok price: 0.3 demo/large: input: unit: mtok price: 1.0 output: unit: mtok price: 3.0 ``` ### 30.3 Spec `reply_benchmark.yaml`: ```yaml benchmark: reply-quality: description: Compare reply generation quality, latency, and cost. dataset: cases: - id: refund_case input: message: I need a refund for a duplicate charge. expected: required_phrase: order id - id: account_case input: message: I cannot log into my account. expected: required_phrase: help run: python: app.benchmarks.reply:run_reply_case variants: reply_v1: label: concise reply factors: model: value: demo/small semantic: llm.model.name optimize: true provider: value: demo semantic: llm.provider tone: value: concise reply_v2: label: fuller reply factors: model: value: demo/large semantic: llm.model.name optimize: true provider: value: demo semantic: llm.provider tone: value: helpful optimize: true score: success: pass: output.contains_required_phrase semantic: result.success role: objective goal: maximize phrase_correctness: exact: actual: output.contains_required_phrase expected: true semantic: quality.correctness role: objective goal: maximize derive: - kind: token_cost pricing: pricing/demo-models.yaml policies: - name: must_pass metric: result.success must_equal: true - name: cost_ceiling metric: money.cost must_less_equal: 0.01 report: leaderboard: show: pass_rate: metric: result.success aggregate: ratio_true avg_correctness: metric: quality.correctness aggregate: mean total_cost: metric: money.cost aggregate: sum avg_latency: metric: time.latency aggregate: mean matrix: metric: quality.correctness compare: reply_v1 -> reply_v2: show: avg_correctness: metric: quality.correctness aggregate: mean total_cost: metric: money.cost aggregate: sum ``` ### 30.4 Run ```bash uv run autobench validate reply_benchmark.yaml uv run autobench run reply_benchmark.yaml --record runs/reply-quality uv run autobench report runs/reply-quality uv run autobench export runs/reply-quality --format markdown --path runs/reply-quality/report.md uv run autobench export runs/reply-quality --format csv --path runs/reply-quality/runs.csv ``` Bu benchmark'ta ayni anda: - success - correctness - token usage - derived cost - latency - policy result - variant comparison - replayable run records toplanir. --- ## 31. Autobench Bir Uygulamaya Nasil Entegre Edilir? Pratik migration sirasi: 1. Mevcut `run_benchmark.py` scriptindeki scenario listesini `Case` haline getir. 2. Model/prompt/tool/config kombinasyonlarini `Variant` ve `FactorValue` haline getir. 3. Eski `run_one_scenario(...)` fonksiyonunu `def run_case(ctx, case)` signature'ina uyarla. 4. Eski manuel metric dictionary'lerini `ctx.metric(...)` veya scorer spec'lerine tasi. 5. Eski timer kodunu `ctx.span(... duration_metric=...)` veya `measure_callable` ile degistir. 6. Eski cost hesaplarini raw token observation + `token_cost` deriver'a ayir. 7. Eski summary table'i `report.leaderboard` ve `report.matrix` ile tanimla. 8. Eski baseline/candidate custom compare kodunu `post_derive: paired_baseline` ile ifade et. 9. Prompt/tool/schema gibi degisen varliklari `track.prompt`, `track.tool`, `track.type` ile takip et. 10. `autobench run --record` ile immutable evidence yaz. Eski script: ```python for scenario in scenarios: result = run_app(scenario) rows.append( { "scenario": scenario.name, "success": result.ok, "latency": result.latency, "cost": result.cost, } ) ``` Autobench task: ```python from autobench import Semantic def run_case(ctx, case): with ctx.span("app_run", duration_metric={"name": "latency"}) as span: result = run_app(case.input) span.set_output(result) ctx.metric("cost", result.cost, semantic_type=Semantic.MONEY_COST, unit="usd") ctx.outcome(result.ok) return {"ok": result.ok, "answer": result.answer} ``` Autobench YAML: ```yaml benchmark: app-benchmark: cases: cases.yaml run: python: app.benchmarks:run_case variants: default: {} score: success: pass: output.ok semantic: result.success report: leaderboard: show: pass_rate: metric: result.success aggregate: ratio_true total_cost: metric: money.cost aggregate: sum avg_latency: metric: time.latency aggregate: mean ``` --- ## 32. Hata Ve Edge Case Davranislari ### 32.1 Task import edilemezse Spec: ```yaml run: python: missing.module:run_case ``` Autobench run'u structured error ile isaretler. CLI hata detayini Rich panelde gosterir. Replay icin task import gerekmedigi icin eski kayitlar bundan etkilenmez. ### 32.2 Scorer path bulunamazsa ```yaml score: bad: value: output.not_here semantic: quality.score ``` Scorer `ScoreRecord.error` uretir. Optional degilse evaluation fail olabilir. Optional scorer: ```yaml score: optional_debug_metric: value: output.debug.score semantic: quality.score optional: true ``` ### 32.3 Cost input eksikse `token_cost` deriver gerekli semantic input'lari bulamazsa cost uydurmaz: ```text token_cost_missing_inputs ``` Diagnostic observation yazar. ### 32.4 Paired baseline match yoksa Candidate run icin baseline match bulunamazsa: ```text paired_baseline_unavailable ``` Diagnostic observation yazar. `missing: skip` dersen sessizce atlar. ### 32.5 Policy metric yoksa Policy result fail olur ve reason: ```text missing_metric ``` --- ## 33. Ne Zaman Task Observation, Ne Zaman Scorer? Kural basit: Task observation: - runtime'da zaten dogal olarak ortaya cikan sey - token usage - latency - retrieved doc count - selected model - tool call count - raw confidence - artifact/trace/debug data Scorer: - output'u expected ile karsilastiran sey - pass/fail - exact match - schema validity - rubric - judge - expected tool action correctness Deriver: - var olan metric/factor'lerden hesaplanan sey - token -> cost - latency baseline/candidate -> speedup - candidate-baseline -> delta Policy: - release gate - "quality >= 0.8" - "cost <= 0.002" - "success must be true" Bu ayrim temiz tutulursa rapor ve autoptimize icin evidence daha guvenilir olur. --- ## 34. Autobench'in Simdiki Sinirlari Bu dosya mevcut durumu anlatiyor. Su an olmayan seyleri de net soylemek lazim: - Chart/image export yok. - Dashboard yok. - Hosted report sharing yok. - Full OpenTelemetry bridge yok. - Logfire/DataDog export yok. - Full autoptimize yok. - Full pydantic-gepa optimizer pipeline Autobench core icinde yok. - Causal attribution engine yok. - Distributed execution yok. - Remote dataset URL loading yok. - YAML icinde inline Python expression yok. Bunlarin bazilari bilerek yok. Ornegin YAML icinde: ```yaml value_factory: len(result.output) ``` gibi expression calistirmak v0.1 yuzeyinde yoktur. YAML shareable ve guvenli kalir. Custom logic gerekiyorsa Python scorer veya task fonksiyonu yazilir. --- ## 35. Sonuc Autobench bugun sunu sagliyor: ```text one-off benchmark script -> YAML-first experiment definition -> semantic observations -> scored/derived/policy-checked runs -> immutable evidence records -> replayable reports -> optimization-ready data ``` Framework'un cekirdek iddiasi sudur: > Kullanici her uygulama icin bastan benchmark runner yazmak zorunda kalmamali. > Case, variant, task, metric, score, derivation, policy, record ve report > primitive'leri bir framework tarafindan saglanmali. AI sistemleri icin bu su demek: - model karsilastirma - prompt version etkisi - tool call kalitesi - structured output validity - token/cost tracking - latency/cost/quality tradeoff - replayable evidence - ileride autoptimize icin candidate feedback AI disi sistemler icin bu su demek: - performans benchmark'i - correctness regression - algorithm variant karsilastirmasi - policy gate - cost/latency/throughput raporu - baseline/candidate speedup Autobench'in su anki hali production-ready bir platform degil; ama dogru primitive'leri olan typed, semantic, replayable bir benchmark evidence layer'dir. --- ## Capability Map Canonical page: https://vcoderun.github.io/autobench/capabilities/ # Capability Map This page is the inventory of what Autobench owns today. Every public feature belongs to one of the layers below; application-specific behavior stays in tasks, scorers, adapters, and metric packs. ## End-To-End Lifecycle ```text BenchmarkSpec -> Dataset x Variants -> BenchmarkPlan -> Task(ctx, case) -> Observations + Spans + Artifacts + Errors -> Scores + Derived Metrics + Policies -> Cross-run Derivation -> Immutable RunRecord / ExperimentRecord -> Replay -> Report -> Compare -> Export -> Optimization Feedback ``` The same lifecycle is available through the YAML DSL, Python models, the `Benchmark` builder, and the CLI. YAML is the portable authoring format; Python remains the extension surface for application execution and custom evaluation logic. ## Definition And Data | Capability | What it provides | | --- | --- | | `BenchmarkSpec` | Validated benchmark metadata, dataset, task, variants, scoring, derivation, policies, and reports | | Dataset | Inline cases, file-backed datasets, glob-backed case files, defaults, tags, metadata, attachments, and versions | | Cases | Arbitrary input and expected payloads with stable IDs and artifact references | | Variants | Named factor combinations with labels, semantic types, and `optimize` hints | | Generated cases | Production-sample conversion, provenance, review status, reasons, and generation batches | | YAML schemas | Versioned JSON schemas and `yaml-language-server` headers for completion and validation | | Source discovery | Hash collection for specs, datasets, pricing files, task modules, and scorer modules | See [Datasets And Variants](datasets-and-variants.md) and [YAML Spec](yaml-spec.md). ## Planning And Execution | Capability | What it provides | | --- | --- | | Matrix planning | Deterministic case x variant expansion and stable run IDs | | Task runtime | Sync and async Python callables with `ctx` first and `case` second | | Concurrency | Bounded async execution while preserving deterministic result ordering | | Failure isolation | One task, scorer, derivation, or policy failure does not erase other runs | | Progress events | Typed lifecycle events for runners and future UI integrations | | Optional Pydantic Evals bridge | Internal conversion to Pydantic Evals-compatible case and dataset payloads | See [Tasks And Runtime](tasks-and-runtime.md). ## Evidence Collection | Capability | What it provides | | --- | --- | | Observations | Metrics, factors, events, diagnostics, artifacts, roles, units, directions, tags, and sources | | Semantic registry | Canonical semantic types, aliases, parent relationships, and custom extensions | | Projection | Source precedence and duplicate detection for one canonical metric view | | Context spans | Nested agent, LLM, tool, retriever, parser, workflow, and custom spans | | Automatic duration | Span timing and optional duration metrics owned by the runtime | | Artifacts | Structured values and files materialized outside the main record payload | | Errors | Structured task, scorer, trace, and policy errors with traceback capture | | Measurement | Warmup, repetitions, time budgets, samples, median, p95, standard deviation, and noise | See [Observations And Semantics](observations-and-semantics.md) and [Instrumentation And Traces](instrumentation-and-traces.md). Native Pydantic AI, OpenAI, OpenAI Agents, and HTTPX integrations can be selected through typed Python settings or the YAML `instrumentation` section. They emit ABP directly, compose across framework/client/transport layers, preserve streaming lifecycle, and remain optional for replay. See [Native Instrumentation](native-instrumentation.md). ## Scoring And Constraints | Scorer | Purpose | | --- | --- | | `output` | Project an output path into a semantic score | | `pass_fail` | Turn a boolean output path into a pass/fail score | | `exact` | Compare actual and expected paths | | `schema` | Validate output against a schema/model | | `python` | Run a sync or async custom scorer using `ScoringCall` | | `expected_action` | Evaluate action/tool selection, arguments, or sequence from spans | Scores declare semantic type, unit, direction, role, and optional failure behavior. Policies add typed requirements including equality, membership, numeric bounds, and inclusive ranges. See [Scoring And Derivation](scoring-and-derivation.md) and [Agentic Evaluation](agentic-evaluation.md). ## Derivation And Cost | Capability | What it provides | | --- | --- | | Token cost | Derive `money.cost` from input/output tokens and normalized model/provider factors | | Pricing DSL | Static YAML pricing, aliases, provider maps, cache prices, token tiers, and model normalization | | Price sources | Optional llm-prices and genai-prices importers that normalize external data into `PricingTable` | | Paired baseline | Per-case or factor-matched speedup, delta, percent change, diagnostics, and verdicts | | Comparison classifier | Improved, regressed, unchanged, or inconclusive outcomes with relative noise thresholds | External price sources are convenience importers, not runtime dependencies or Autobench's source of truth. A local pricing YAML remains fully supported. ## Agentic Evidence Autobench records agent behavior without requiring OpenTelemetry: - typed trace envelopes and nested span records - expected tool/action selection, argument, and sequence checks - span selectors by kind, name, tag, path, or semantic type - Pydantic AI usage normalization - metric packs for agentic, structured-output, LLM-usage, and performance defaults - compact feedback records for optimization systems See [Agentic Evaluation](agentic-evaluation.md). ## Asset Lineage The tracking registry understands: - text prompts from inline text or files - arbitrary assets and configuration values - callable tools, signatures, parameters, docs, and return types - Pydantic models, standard dataclasses, and typed classes - field names, annotations, descriptions, aliases, defaults, requirements, constraints, and examples - source hashes, structured-schema hashes, versions, parent versions, and diffs - persistent human-readable YAML asset histories Decorators preserve the original callable or class type so tracking does not degrade static typing. See [Asset Tracking](asset-tracking.md). ## Records, Replay, And Analysis | Capability | What it provides | | --- | --- | | `RunRecord` | Immutable case x variant evidence including output, scores, observations, spans, factors, assets, artifacts, and errors | | `ExperimentRecord` | Plan, environment, semantic registry, report config, source hashes, and run paths | | Replay | Load records without importing task or scorer modules | | Rich terminal reports | Status, variant configuration, leaderboard, run metrics, case matrix, comparisons, and distributions | | Exports | Human-readable YAML summary, CSV run projection, and Markdown report | | Optimization feedback | Failure category, score, reasons, factors, asset versions, and selected evidence | See [Recording And Reporting](recording-and-reporting.md). ## Ownership Boundaries Autobench deliberately does not own: - application or model execution - hosted tracing or observability storage - model-specific pricing as an always-current service - causal claims from confounded comparisons - optimizer search strategies or candidate promotion - large catalogs of domain-specific LLM judges Tasks and adapters own application execution. Optional integrations may import traces, pricing, or evaluator results, but the core contract remains semantic, generic, and replayable. --- ## Core Concepts Canonical page: https://vcoderun.github.io/autobench/concepts/ # Concepts ## BenchmarkSpec The YAML source of truth. It defines: - benchmark metadata - dataset and cases - task target - variants and factors - scoring - derivation and post-derivation - report configuration ## Case A single benchmark input with optional expected output, metadata, tags, and attachments. Cases are the unit of replay and comparison. ## Variant A concrete factor set for a run. Variants represent the changing parts of the system: - prompt version - model - provider - policy - tool version ## Observation The atomic evidence unit recorded during execution. Observations can be: - metrics - factors - events - artifacts Observations may carry a semantic type such as `quality.correctness`, `money.cost`, or `llm.tokens.input`. ## Trace Envelope A structured trace attached to a run. Trace envelopes preserve agent/workflow spans without making OpenTelemetry or any hosted tracing platform a core dependency. Trace spans can be typed as: - `agent` - `llm` - `tool` - `retriever` - `parser` - `workflow` - `custom` Autobench converts useful trace fields into semantic observations, such as token usage, model/provider factors, duration, and span errors. Large raw trace payloads should be stored as artifacts instead of being embedded directly into `run.yaml`. ## Expected Action An expected action describes behavior a system should perform during a run. It is generic enough for agent tools, retrievers, workflow steps, or other component calls. Expected actions support: - target matching - input subset matching - output matching - ordered sequence checks - optional vs required actions The built-in `expected_action` scorer can emit `agent.tool.selection.correctness`, `agent.tool.argument.correctness`, and `agent.tool.sequence.correctness` style metrics without requiring an LLM judge. ## Metric Pack A metric pack is an optional bundle of semantic registry entries, scorer defaults, report metrics, and feedback extractors. Metric packs keep Autobench from becoming a large class-per-metric catalog. Core owns evidence, records, and execution; packs provide domain defaults such as `agentic`, `structured_output`, `llm_usage`, and `performance`. ## Score A score is a structured evaluation result produced by the scoring layer and projected into observations with score precedence. ## Derived Metric A metric computed from observed inputs. Example: token usage plus model/provider factors become `money.cost`. ## Post-Derivation Cross-run derivation that needs the full experiment result. Example: paired-baseline latency speedup. ## Run Record An immutable YAML snapshot of one case x variant execution, including: - task output - observations - spans - scores - factors - tracked asset versions - artifacts - errors ## Report A replay-time view over recorded runs. Reports aggregate semantic metrics into: - leaderboards - case matrices - comparisons - metric distributions ## Optimization Feedback Autobench can compact failed scores, task errors, span errors, policy violations, factors, and asset versions into feedback records. These records are designed for optimization systems such as pydantic-gepa and autoptimize, so they do not need to scrape raw reports or infer failure categories from terminal output. --- ## Datasets And Variants Canonical page: https://vcoderun.github.io/autobench/datasets-and-variants/ # Datasets And Variants Autobench expands a dataset against variants to create a deterministic run matrix. Cases describe what is evaluated; variants describe what changes. ## Cases Each `Case` has a stable ID and may carry arbitrary input, expected output, metadata, tags, and attachments: ```python from autobench import Case case = Case( id="refund-request", input={"message": "I need a refund for order 42"}, expected={"route": "billing", "priority": "normal"}, metadata={"tenant": "demo"}, tags=["routing", "smoke"], ) ``` Inputs and expected values are intentionally generic. They can be strings, mappings, Pydantic models serialized by the task, structured multimodal references, or domain-specific payloads. Attachments use `ArtifactRef` values when a case depends on external material. ## Dataset Sources Cases may be authored inline: ```yaml dataset: version: v1 cases: - id: refund-request input: message: I need a refund expected: route: billing ``` Or loaded relative to the benchmark file: ```yaml dataset: source: file://datasets/cases.yaml version: v1 ``` File-backed datasets use the same DSL representation. Glob-backed sources can combine separate case files while duplicate case IDs remain validation errors. ## Case Defaults Defaults reduce repeated metadata without hiding the final case payload: ```yaml dataset: defaults: metadata: locale: en-US tags: [regression] cases: - id: ticket-1 input: {message: Reset my password} tags: [authentication] ``` Mapping values are merged, tags are deduplicated, and explicit scalar case values override defaults. ## Variants And Factors A variant is one concrete factor set: ```yaml variants: baseline: label: Current production route factors: model: value: openrouter:openai/gpt-5.6-luna semantic: llm.model.name optimize: true prompt_version: value: route-v3 semantic: prompt.version optimize: true temperature: 0 ``` `value` is the runtime value. `semantic` tells downstream consumers what the factor means. `optimize` is a hint that the factor is a candidate optimization axis; Autobench records it but does not choose search strategies. The Python form is equivalent: ```python from autobench import FactorValue, Semantic, Variant variant = Variant( id="baseline", label="Current production route", factors=[ FactorValue( name="model", value="openrouter:openai/gpt-5.6-luna", semantic_type=Semantic.LLM_MODEL_NAME, optimize=True, ), FactorValue(name="temperature", value=0), ], ) ``` Tasks read factors through `ctx.factor(name)`. Factors are also copied into RunRecords and report variant-configuration tables. ## Generated And Production Cases The data helpers preserve where generated examples came from: - `ProductionSample` models a source sample and review state. - `sample_to_case` and `samples_to_cases` convert samples without losing provenance. - `mark_generated_case` records generation metadata. - `generated_batch_from_cases` creates a `GeneratedCaseBatch` with policy and source details. This layer is intentionally not a synthetic-data generator. It defines the evidence contract so a generator, production sampler, or review system can supply cases consistently. ## Identity And Reproducibility - Case IDs and variant IDs must be unique. - Dataset content hashes depend on normalized content rather than filesystem location. - Matrix order is deterministic. - Run IDs are stable for a given plan position, case, and variant. - Dataset version may also be emitted as `dataset.version` semantic evidence. --- ## Tasks And Runtime Canonical page: https://vcoderun.github.io/autobench/tasks-and-runtime/ # Tasks And Runtime The task is the only application-specific execution boundary required by Autobench. It receives a runtime context and a case, invokes the subject, records evidence, and returns the output that scorers evaluate. ## Task Contract ```python from autobench import Case, RunContext def run_case(ctx: RunContext, case: Case) -> dict[str, object]: model = ctx.factor("model") result = call_application(case.input, model=model) ctx.outcome(result.ok) return {"answer": result.answer, "ok": result.ok} ``` The positional contract is always `task(ctx, case)`. Tasks may be synchronous or asynchronous: ```python async def run_case(ctx: RunContext, case: Case) -> dict[str, object]: result = await call_application(case.input) return {"answer": result.answer} ``` YAML resolves the callable relative to the benchmark file before falling back to import paths: ```yaml run: python: benchmark_tasks:run_case ``` ## RunContext `RunContext` owns evidence for one case x variant run: | Method | Use | | --- | --- | | `factor(name)` | Read a variant factor | | `span(...)` | Open a timed nested operation | | `metric(...)` / `metrics(...)` | Record one or many metrics | | `factor_observation(...)` | Record a runtime-discovered factor | | `event(...)` | Record a discrete event | | `diagnostic(...)` | Record non-objective diagnostic evidence | | `outcome(...)` | Record semantic run success | | `check(...)` | Record a boolean correctness check with an optional reason | | `record_measurement(...)` | Record summary statistics and optional sample artifact | | `artifact(...)` | Attach a structured or file-like payload | | `error(...)` | Attach a structured error without losing collected evidence | | `attach_tracked_asset(...)` | Bind a tracked asset version to the run | Context evidence remains available even when the task raises. The runtime captures the exception, preserves observations and artifacts already emitted, and records a structured error. ## Matrix Execution `build_benchmark_plan` validates and counts the matrix before execution. `expand_matrix` produces one `MatrixRunSpec` per case x variant pair. The CLI renders the same plan during validation. ```bash autobench validate autobench.yaml autobench run autobench.yaml --concurrency 4 --record runs/latest ``` Concurrency bounds the number of active runs. Result ordering stays deterministic even when task completion order differs. ## Failure And Status Model Autobench separates three status layers: - `TaskStatus`: whether application execution completed, failed, or was skipped. - `EvaluationStatus`: whether scoring and constraints completed. - `RunStatus`: final passed, failed, errored, or skipped state. This distinction prevents a policy failure from looking like an application exception and lets reports separate execution reliability from evaluation quality. ## Progress Events `ProgressEvent` and `ProgressEventKind` provide typed lifecycle notifications. Known event fields remain stable while event-specific data is carried in the payload. This is the extension surface for terminal progress, service runners, and future UIs without coupling the core runtime to one frontend. ## Python Builder The builder compiles to the same `BenchmarkSpec` used by YAML: ```python from autobench import Benchmark, Case, FactorValue, PassFailScorer, Semantic, Variant result = ( Benchmark("routing") .dataset([Case(id="refund", input={"message": "Refund order 42"})]) .variants( [ Variant( id="baseline", factors=[FactorValue(name="route", value="v1")], ) ] ) .task("benchmark_tasks:run_case") .scoring( [ PassFailScorer( name="success", path="output.ok", semantic_type=Semantic.RESULT_SUCCESS, ) ] ) .run() ) ``` Use YAML for portable benchmark definitions and the builder when a Python application needs to compose specs programmatically. Both execute through the same planner and runtime. --- ## Observations And Semantics Canonical page: https://vcoderun.github.io/autobench/observations-and-semantics/ # Observations And Semantics An observation is Autobench's atomic evidence unit. Raw names remain useful to humans, while semantic types make evidence portable across applications, reports, and optimizers. ## Observation Model An `Observation` carries: - stable ID and local name - kind: metric, factor, event, diagnostic, or artifact - value and optional unit - semantic type - optimization direction and role - source and optional span ID - tags, case ID, and variant ID ```python from autobench import Direction, ObservationRole, Semantic ctx.metric( "answer_accuracy", 0.94, semantic_type=Semantic.QUALITY_CORRECTNESS, direction=Direction.MAXIMIZE, role=ObservationRole.OBJECTIVE, ) ``` The local name may be `answer_accuracy`, `judge_score`, or `coverage`; the semantic type tells the framework whether those values share meaning. ## Built-In Semantic Families | Family | Examples | | --- | --- | | LLM | `llm.tokens.input`, `llm.tokens.output`, `llm.request.count`, `llm.model.requested`, `llm.model.response`, `llm.provider.name` | | Cost | `money.cost`, `serving.cost`, `optimization.cost`, `lifetime.cost` | | Time | `time.latency`, `time.first_chunk`, `time.critical_path` | | Result | `result.success` | | Quality | `quality.score`, `quality.correctness`, `coverage.ratio` | | Agent | task completion, plan quality/adherence, step efficiency, tool selection/arguments/sequence, output correctness | | Assets | `prompt.version`, `agent.tool.version`, `agent.version`, `dataset.version` | | Operations | count, maximum depth/fan-out, incomplete work, parallelism, retries, recovered retries, first-attempt success | | Workflow | validation failures, approval count/wait, tool-call success/failure, message growth, evidence-reference counts | `Semantic` exposes completion-friendly constants. `SemanticType` remains extensible so domain metrics can use names such as `retrieval.recall` or `business.conversion`. ## Registry And Aliases `SemanticRegistry` stores definitions, aliases, and parent relationships. A custom registry can be embedded in a benchmark spec and is merged with built-ins: ```yaml semantic_registry: version: 1 types: business.conversion: description: Whether the workflow produced a qualified conversion. parent: result.success unit: boolean aliases: conversion: business.conversion ``` Parent relationships let a query request a broad semantic category while preserving specific metrics. Aliases prevent local naming differences from fragmenting evidence. ## Roles And Directions Roles describe how a metric participates in evaluation: - objective: something to optimize - constraint: something that must remain acceptable - diagnostic: explanatory evidence Directions are `maximize` or `minimize`. Factors, events, and artifacts cannot declare an optimization direction because they are not outcomes. ## Sources And Projection The same semantic metric can be emitted by a task, scorer, deriver, policy, or adapter. Raw observations are never discarded. Projection chooses a canonical value using explicit source priority and ABP accounting scope. A derived aggregate summary is preferred to same-source direct measurements for single-value reporting, while direct observations remain queryable. Logical operation IDs correlate equivalent framework/client evidence; equal-priority disagreements are marked ambiguous instead of silently picking one. Use `ObservationQuery` for raw or projected lookup and `filter_observations` for selectors such as semantic type, role, source, or span. ```python from autobench import ObservationQuery query = ObservationQuery(observations=list(result.observations)) costs = query.values("money.cost", projected=False) ``` Reports, policies, and derivation use this semantic projection layer rather than relying on local metric names. ## Metric Packs A `MetricPack` bundles reusable semantic defaults without forcing every metric into core: - semantic registry additions - scorer factory references - default report metrics - feedback extractors Built-in packs cover `agentic`, `structured_output`, `llm_usage`, and `performance`. Applications can register their own packs through `MetricPackRegistry` while keeping the RunRecord contract unchanged. --- ## Asset Tracking Canonical page: https://vcoderun.github.io/autobench/asset-tracking/ # Asset Tracking Benchmarks need to know which prompt, tool, schema, or configuration produced each result. Autobench tracking assigns content-derived versions, captures structured metadata, persists history, and binds exact asset versions to RunRecords. ## Prompts And Text Assets Track inline text: ```python from autobench import track SYSTEM_PROMPT = track.prompt( name="support_system_prompt", text="Route the request to billing, account, or technical support.", ) ``` Or load it from a file: ```python SYSTEM_PROMPT = track.prompt( name="support_system_prompt", source="prompts/support.md", ) ``` `TrackedPrompt.raw` returns the text, and `str(SYSTEM_PROMPT)` provides the same value for APIs that expect a string. File-backed prompts retain their source path and source hash. ## Tools `@track.tool` preserves the callable's exact signature and return type while collecting tool metadata: ```python from typing import Literal from autobench import track @track.tool def route_ticket( queue: Literal["billing", "account", "technical"], priority: int = 1, ) -> bool: """Route a ticket to a support queue.""" return priority > 0 ``` The resulting `ToolAsset` records: - qualified name and docstring - parameter names, kinds, annotations, defaults, and requirements - return annotation - source path and source hash - structured parameter schema - semantic type and version lineage Annotations are normalized by structure rather than alias spelling. If the contents of a `Literal`, union, generic, model, or referenced type change, the asset hash changes even when the alias name stays the same. ## Pydantic Models, Dataclasses, And Classes ```python from dataclasses import dataclass from typing import Literal from autobench import track from pydantic import BaseModel, Field @track.type class Car(BaseModel): make: Literal["audi", "bmw", "mercedes"] model: str = Field(examples=["a3", "320i"]) year: int = Field(gt=0) @track.dataclass(frozen=True, slots=True) class CarRequest: make: Literal["audi", "bmw", "mercedes"] model: str year: int ``` Pydantic models are hashed from normalized JSON Schema plus source identity. Standard dataclasses use dataclass field definitions and resolved annotations. Other typed classes use resolved class annotations, inspectable signatures, and source hashes. `TypeAsset` and `FieldAsset` preserve field names, resolved annotations, descriptions, examples, aliases, defaults, required state, and relevant constraints. ## Composing Another Class Decorator When `@track.type` above a class-transforming decorator gives poor type-checker inference, use `track.decorate_type`: ```python from dataclasses import dataclass from autobench import track @track.decorate_type(dataclass, frozen=True, slots=True) class Request: value: str ``` The decorator and its normalized arguments are stored as asset metadata. `track.dataclass(...)` is the typed convenience form for the standard dataclass decorator. ## Arbitrary Assets Use `track.asset` for configurations, policies, routing tables, or other application components: ```python @track.asset(kind="routing_policy", name="enterprise_routing") def route_policy(ticket): return "priority" if ticket["enterprise"] else "standard" ``` The decorator returns the original object unchanged. Callables use source and signature metadata; manual `version`, `hash`, `source_path`, `parent_version`, and metadata values are available when automatic identity is not enough. ## Versions, Diffs, And Persistence `TrackingRegistry` keeps current assets and version history in memory during execution. Persist it with: ```python from pathlib import Path from autobench import track track.write_assets(Path(".autobench/assets")) ``` The YAML history contains an index plus one file per asset. Every new version links to its parent when available and stores a human-readable diff from the previous serialized state. Source changes, schema changes, decorator options, and metadata changes therefore remain reviewable. ## Binding Assets To Runs ```python def run_case(ctx, case): ctx.attach_tracked_asset(SYSTEM_PROMPT) ctx.attach_tracked_asset(route_ticket) ctx.attach_tracked_asset(Car) return execute(case.input) ``` The exact `AssetVersion` values are copied into the RunRecord. Reports and optimization feedback can then relate metric changes to prompt, tool, or output-schema versions without guessing from source control state. --- ## Agentic Evaluation Canonical page: https://vcoderun.github.io/autobench/agentic-evaluation/ # Agentic Evaluation Autobench evaluates agents as traced systems rather than treating only the final text as evidence. The same primitives also work for workflow engines, retrievers, and tool-using applications. ## Record Agent Behavior ```python from autobench import Semantic, SpanKind def run_case(ctx, case): with ctx.span("support_agent", kind=SpanKind.AGENT, input=case.input) as agent: with ctx.span( "lookup_user", kind=SpanKind.TOOL, input={"user_id": case.input["user_id"]}, ) as tool: profile = lookup_user(case.input["user_id"]) tool.set_output(profile) answer = compose_answer(profile, case.input["message"]) agent.set_output(answer) agent.metric( "task_completed", True, semantic_type=Semantic.AGENT_TASK_COMPLETION, ) return answer ``` Spans preserve selection, arguments, output, order, duration, errors, tags, and hierarchy. ## Declare Expected Actions Cases can use generic `actions` or the tool-oriented `tool_calls` compatibility shape: ```yaml cases: - id: refund input: user_id: u1 message: Refund order 42 expected: actions: - id: lookup kind: tool target: lookup_user input: user_id: u1 order: 1 required: true ``` Expected input matching is subset-based, so a tool may receive additional nonessential arguments. Actions may also declare expected output, tolerance metadata, optional status, and explicit order. ## Score Selection, Arguments, And Sequence ```yaml score: tool_selection: expected_action: metric: selection observed_kind: tool span: kind: tool semantic: agent.tool.selection.correctness goal: maximize tool_arguments: expected_action: metric: arguments observed_kind: tool span: kind: tool semantic: agent.tool.argument.correctness goal: maximize tool_sequence: expected_action: metric: sequence observed_kind: tool span: kind: tool semantic: agent.tool.sequence.correctness goal: maximize ``` These scorers are deterministic and do not require an LLM judge. They produce normal scores and semantic observations, so policies and reports consume them like any other metric. ## Span Selection `SpanSelector` filters spans by: - kind - name - tags - nested path - emitted semantic type Selectors can be composed with positive and negative report/evaluation filters. A scorer receives the selected spans through `ScoringCall`, allowing custom component-level evaluators without parsing raw traces. ## Agentic Semantic Types Built-in semantics include: - task completion and goal accuracy - plan quality and plan adherence - step efficiency and orchestration quality - tool name and version - tool selection, argument, and sequence correctness - tool-call quality - output correctness and structure validity - agent version and serving volume Applications may add more specific child semantics through the registry. ## Metric Packs The `agentic` metric pack contributes standard semantic definitions and report defaults. Metric packs are optional: they provide conventions, not a required agent SDK. A custom agent runtime can emit the same evidence through spans or a trace adapter. ## Optimization Feedback `build_feedback_records` compacts run evidence into one record per case. It captures: - score and evaluator reasons - task, scorer, policy, and span errors - `failure_category` only when a failure exists - factor values and tracked asset versions - selected observations and trace context `build_optimization_feedback_input` packages those records with benchmark identity and semantic context. pydantic-gepa or autoptimize can consume this structured evidence without scraping Rich tables or replay YAML. Autobench reports association and comparison evidence; it does not claim causal attribution when multiple factors changed together. Controlled experiment planning belongs to the optimizer layer. --- ## Scoring And Derivation Canonical page: https://vcoderun.github.io/autobench/scoring-and-derivation/ # Scoring And Derivation Scorers evaluate one run. Derivers compute new metrics from collected evidence. Post-derivers work across runs after the complete experiment exists. Policies turn semantic metrics into explicit requirements. ## Scoring Contract Every scorer declares: - a local score name - semantic type - optional unit - optimization direction - role: objective, constraint, or diagnostic - whether scorer failure is optional Scores are stored as `ScoreRecord` values and projected into observations with score-source precedence. The original task observations remain available. ## Output Metric Project an output value directly: ```yaml score: coverage: value: output.coverage semantic: coverage.ratio goal: maximize role: objective ``` Use this when the task already computes a trustworthy metric. ## Pass/Fail ```yaml score: success: pass: output.ok semantic: result.success role: constraint ``` The path must resolve to a boolean-like success value. ## Exact Match ```yaml score: route_correctness: exact: actual: output.queue expected: case.expected.queue semantic: quality.correctness goal: maximize ``` Paths can address `output`, `case.input`, `case.expected`, factors, and structured values. ## Schema Validation `SchemaScorer` validates a selected output path against a JSON Schema mapping. It is appropriate for contracts where structural validity is separate from domain correctness. ```python from autobench import SchemaScorer, Semantic scorer = SchemaScorer( name="output_schema", path="output", schema={ "type": "object", "required": ["customer_name", "id"], "properties": { "customer_name": {"type": "string"}, "id": {"type": "string"}, }, }, semantic_type=Semantic.AGENT_OUTPUT_STRUCTURE_VALIDITY, ) ``` ## Python Scorers Custom scorers receive `ScoringCall`, not loose callback dictionaries: ```python from autobench import ScoreRecord, ScoringCall def field_accuracy(call: ScoringCall) -> ScoreRecord: expected = call.case.expected output = call.output fields = ("name", "id", "pocket_id") matches = sum(output[field] == expected[field] for field in fields) return ScoreRecord( name="field_accuracy", semantic_type="quality.field_accuracy", value=matches / len(fields), ) ``` `ScoringCall` exposes the case, variant, task output/result, observations, spans, and selected spans. Python scorers may be sync or async. Optional scorers record errors without failing the run. ## Expected Actions `ExpectedActionScorer` deterministically evaluates action/tool selection, arguments, or ordered sequence from spans. See [Agentic Evaluation](agentic-evaluation.md). ## Dotted Paths `resolve_dotted_path` is the shared structured-path resolver used by built-in scorers. Missing paths produce explicit scorer errors instead of silently returning `None`. ## Per-Run Derivation `derive` runs after task observations and scores are available for one run. `TokenCostDeriver` is the built-in per-run deriver. ```yaml derive: - kind: token_cost pricing: file://pricing/models.yaml output: name: request_cost semantic_type: money.cost unit: usd direction: minimize role: constraint ``` By default it reads: - `llm.tokens.input` - `llm.tokens.output` - `llm.provider` - `llm.model.name` Input semantics and output metadata can be overridden through `TokenCostInputs` and `DerivedMetricOutput` in the Python API. Unknown models, missing usage, missing rates, and ambiguous inputs produce diagnostics; Autobench does not invent a zero cost. ## Pricing DSL Pricing is normalized into a `PricingTable` keyed by stable model IDs. Provider-specific aliases allow input forms such as `provider:model`, `provider/model`, or application-specific model slugs to resolve to the same entry. ```yaml pricing: version: 1 provider: openai models: openai/gpt-demo: aliases: [openai:gpt-demo, gpt-demo] input: unit: mtok price_per_million_tokens: 1.0 output: unit: mtok tiers: - up_to_tokens: 100000 price_per_million_tokens: 4.0 - price_per_million_tokens: 6.0 cache_read: unit: mtok price_per_million_tokens: 0.1 ``` Supported fields include input, output, cache-read, and cache-write prices plus token-count tiers. `StaticPriceSource`, `LLMPricesSource`, and `GenAIPricesSource` only import external price data into this model. They do not make an external catalog authoritative at runtime. ## Paired Baseline Post-Derivation `post_derive` has access to the full experiment: ```yaml post_derive: - kind: paired_baseline baseline_variant: baseline match_on: - kind: case_id - kind: factor name: workload.size metric: time.latency formula: baseline_over_candidate include_baseline: true output: name: speedup semantic_type: performance.speedup unit: ratio direction: maximize role: objective ``` Formulas: - `baseline_over_candidate` - `candidate_over_baseline` - `candidate_minus_baseline` - `baseline_minus_candidate` - `percent_change_from_baseline` Matching supports case IDs and factor keys. Missing matches, nonnumeric metrics, absent metrics, and zero division can be skipped or recorded as diagnostics. Relative-noise thresholds and `ComparisonVerdictSpec` can emit improved, regressed, unchanged, or inconclusive verdicts. These are controlled comparisons, not automatic causal claims. ## Policies Policies evaluate projected semantic values and append `PolicyResult` evidence: ```yaml policies: - name: request-must-succeed metric: result.success must_equal: true - name: cost-cap metric: money.cost must_less_equal: 0.001 - name: acceptable-latency metric: time.latency must_between: min: 0 max: 500 inclusive: true ``` Each policy declares exactly one requirement: - `must_equal` / `must_not_equal` - `must_greater` / `must_greater_equal` - `must_less` / `must_less_equal` - `must_in` / `must_not_in` - `must_between` A failed constraint can change final run status while preserving the successful task output and all evidence that explains the decision. ## Repeated Measurement `measure_callable` avoids repeating warmup and sampling loops in benchmark tasks: ```python from autobench import MeasurementBudget, measure_callable measurement = measure_callable( lambda: search(case.input["items"], case.input["query"]), budget=MeasurementBudget(warmup=3, repetitions=20, max_seconds=2.0), ) ctx.record_measurement("search", measurement) ``` `Measurement` includes samples, count, min, max, mean, median, p95, standard deviation, and relative noise. A custom timer can measure accelerators or remote systems without adding domain-specific logic to Autobench. --- ## Recording And Reporting Canonical page: https://vcoderun.github.io/autobench/recording-and-reporting/ # Recording And Reporting Recording turns an in-memory experiment into portable, immutable evidence. Replay and analysis use those records without executing the application again. ## Record Layout ```bash autobench run autobench.yaml --record runs/support-routing ``` The directory contains: ```text runs/support-routing/ experiment.yaml summary.yaml cases///run.yaml artifacts/... ``` Paths are stable and artifact references are relative so the directory can be moved or archived. Recording is append-only: an existing run payload is never silently replaced. ## RunRecord One `RunRecord` represents one case x variant execution: - record, run, experiment, benchmark, case, and variant IDs - final, task, and evaluation statuses - complete case snapshot and task output - observations and scores - canonical ABP trace, including signals, span graph, measurements, events, links, references, diagnostics, and instrumentation scope provenance - ABP protocol and semantic registry versions - legacy span tree for records created before canonical trace storage - materialized artifacts - factors and tracked asset versions - extraction and source-map replay lineage - structured errors The YAML view groups the data for people rather than dumping internal Pydantic fields. A schema header points editors to the versioned Autobench JSON schema. Small traces remain inline in `run.yaml`. Larger traces are written to `artifacts//trace.yaml`; the RunRecord keeps a relative `ArtifactRef` and a compact trace summary. Trace artifacts have their own versioned JSON Schema header and load back into the same typed `Trace` model. ## ExperimentRecord The experiment-level record stores: - benchmark plan and counts - captured environment metadata - semantic registry - report configuration - normalized benchmark snapshot and hash - hashes of resolved specs, datasets, pricing files, tasks, and scorer modules - relative run paths and status counts This is enough to explain what was planned, which files defined it, and where every run record lives. ## Environment And Source Identity `capture_environment` records reproducibility metadata such as Python, platform, package, and working-environment details. `collect_benchmark_source_files` resolves benchmark dependencies and records content hashes. Source paths are stored portably when possible. Missing optional source files do not erase a run; recording captures what was resolvable at execution time. ## Artifacts `ctx.artifact(name, value)` adds an `ArtifactRef`. During recording, supported values are materialized under `artifacts/` and the RunRecord keeps the relative path, media type, and tags. Use artifacts for: - generated specs and prompts - traces too large for `run.yaml` - measurement samples - model responses and structured debug payloads - Markdown or text reports produced by the subject Artifact path collisions and attempts to overwrite existing payloads are recording errors. ## Replay ```bash autobench replay runs/support-routing ``` Replay loads `ExperimentRecord` and every `RunRecord` into an `ExperimentResult`. It deliberately does not import task or scorer modules, call models, or mutate the original directory. This enables: - offline report regeneration - new exports from old evidence - baseline/candidate comparison after execution - future rescoring into a separate derived experiment - optimization systems consuming stable records Autobench distinguishes three replay modes: - **report replay** reads stored observations without re-extracting evidence - **extraction replay** runs a typed `TraceExtractor` against the immutable ABP trace and creates a derived RunRecord - **canonicalization replay** applies newer source maps to retained source snapshots and creates a separate derived RunRecord Derived records point to the original `run_id`, identify the extractor or source-map versions, and retain the source protocol and semantic registry versions. The original record and trace bytes are never rewritten. Replay resolves trace artifacts only inside the experiment directory and imports neither application task modules nor optional SDK integrations. The default `SignalExtractor` reconstructs canonical observations from stored ABP measurements and events. `SpanExtractor` derives generic topology and workflow evidence, while `UsageExtractor` owns LLM request/token/model accounting. `CompositeExtractor` can run them as one versioned replay processor. Custom extractors implement the typed `TraceExtractor` interface and return observations, diagnostics, and evidence references without mutating the trace. When a newer version of the same extractor is replayed, its observations replace the older version's observations in the new derived record. The previous derived record remains the lineage parent, so extractor evolution is auditable without mixing two versions of one derived metric. ## Rich Reports ```bash autobench report runs/support-routing ``` The terminal report can include: - experiment overview and status counts - variant configuration table with factor values - semantic leaderboards - per-run metric tables grouped by semantic family - case x variant matrices - baseline/candidate factor and metric deltas - metric distributions Reports use projected semantic metrics. They do not depend on application-specific local names. ## Report Configuration ```yaml report: leaderboard: show: accuracy: metric: quality.correctness aggregate: ratio_true total_cost: metric: money.cost aggregate: sum p95_latency: metric: time.latency aggregate: p95 matrix: metric: quality.correctness compare: baseline -> candidate: show: accuracy: metric: quality.correctness aggregate: ratio_true distributions: - name: request_latency semantic_type: time.latency summaries: [min, median, p95, max] ``` Aggregation functions include count, mean, sum, min, max, median, p95, standard deviation, geometric mean, and boolean true ratio. ## Comparison Semantics ```bash autobench compare runs/support-routing --baseline baseline --candidate candidate ``` Comparison pairs runs by case, displays changed factors, aggregates requested semantic metrics, and sets `confounded=true` when multiple relevant factors changed. It reports association and deltas; it does not claim which factor caused the result. Use paired-baseline post-derivation when a per-run derived metric such as speedup must be written back into candidate evidence. ## Exports ```bash autobench export runs/support-routing --format yaml --path report.yaml autobench export runs/support-routing --format csv --path runs.csv autobench export runs/support-routing --format markdown --path report.md ``` - YAML is a human-readable summary projection. - CSV is a flat run-and-metric table for analysis tools. - Markdown is a portable rendered report. The CLI always writes the requested file and then renders a Rich preview. Machine exports never replace immutable source RunRecords. --- ## CLI Canonical page: https://vcoderun.github.io/autobench/cli/ # CLI The CLI is human-first: validation, runs, replay, reports, and comparisons render Rich panels and tables. YAML, CSV, and Markdown are explicit file exports rather than raw terminal dumps. ## Command Summary | Command | Executes tasks? | Requires records? | Purpose | | --- | --- | --- | --- | | `validate` | No | No | Parse, validate, resolve sources, and show the planned matrix | | `run` | Yes | No | Execute a benchmark, optionally persist it, and render results | | `replay` | No | Yes | Reconstruct and display the recorded experiment | | `report` | No | Yes | Render configured analysis views from records | | `export` | No | Yes | Write YAML, CSV, or Markdown and preview it | | `compare` | No | Yes | Compare two recorded variants without claiming causality | | `instrumentation doctor` | No | No | Inspect integration compatibility, capabilities, and capture defaults | | `instrumentation trace` | No | Yes | Summarize ABP trace composition and partial state | References inside a benchmark spec resolve relative to the spec file. ## Commands ### Validate ```bash uv run autobench validate path/to/spec.yaml ``` Validation loads external datasets and referenced configuration, checks duplicate IDs and task requirements, resolves source files, and renders case, variant, and planned-run counts. It does not execute the task target. ### Run ```bash uv run autobench run path/to/spec.yaml --record runs/example ``` Options: - `--concurrency INTEGER`: maximum active runs, default `1`, minimum `1`. - `--record DIRECTORY`: explicit immutable record directory. - `--no-record`: execute and report without persistence. Without either recording flag, Autobench writes under `.autobench///`. ### Replay ```bash uv run autobench replay runs/example ``` Replay does not import tasks, scorers, or application modules. It reconstructs the experiment from the record directory and renders the recorded report configuration. ### Report ```bash uv run autobench report runs/example ``` `report` emphasizes status, variant configuration, leaderboards, per-run metrics, case matrices, comparisons, and distributions. ### Export ```bash uv run autobench export runs/example --format yaml --path runs/example/report.yaml uv run autobench export runs/example --format csv --path runs/example/runs.csv uv run autobench export runs/example --format markdown --path runs/example/report.md ``` ### Compare ```bash uv run autobench compare runs/example --baseline baseline --candidate optimized ``` Both IDs must exist in the recorded experiment. The command shows paired-run count, changed factors, aggregate metric deltas, and a confounding flag. ### Instrumentation Diagnostics ```bash uv run autobench instrumentation doctor uv run autobench instrumentation trace runs/example ``` `doctor` inspects every built-in integration without importing unavailable SDKs. Its Rich tables show target versions, supported status, abstraction layer, hook/patch mechanism, sync/async/streaming support, span and semantic families, capture defaults, extras, and degradation details. `trace` operates only on recorded evidence. It reports case/variant span counts, roots, partial state, diagnostics, span kinds, and instrumentor composition without importing the benchmark task or provider SDKs. ## CLI Behavior - `run` executes the benchmark matrix, optionally records it, and renders Rich summary tables. - `replay`, `report`, `export`, and `compare` operate on recorded evidence. - `instrumentation trace` has the same replay-only dependency boundary. - `report` and `compare` render Rich terminal views instead of dumping Markdown or YAML. - `export` always writes a file and then shows a Rich preview of the exported projection. - default recording paths are placed under `.autobench///`. ## Exit And Error Behavior - Invalid YAML, schema errors, unresolved tasks, recording collisions, and missing records return a nonzero exit code. - User-facing errors include the relevant file and YAML location when available. - A process may complete while individual runs are failed, errored, or skipped; the status table makes those states explicit. - Replay and reporting never fall back to live benchmark execution. ## Typical Workflow ```bash autobench validate autobench.yaml autobench run autobench.yaml --concurrency 4 --record runs/candidate-42 autobench report runs/candidate-42 autobench compare runs/candidate-42 --baseline baseline --candidate candidate autobench export runs/candidate-42 --format csv --path analysis/runs.csv ``` --- ## Protocol And Traces Canonical page: https://vcoderun.github.io/autobench/instrumentation-and-traces/ # Instrumentation And Traces Autobench supports four collection styles that can be mixed in one run: 1. Explicit `RunContext` and `Span` calls inside a task. 2. Lightweight method instrumentation for existing application classes. 3. Trace-envelope adapters for an external agent or workflow runtime. 4. Native Pydantic AI, OpenAI, OpenAI Agents, and HTTPX instrumentors configured from Python or YAML. OpenTelemetry is not a core dependency. Future OTLP bridges can export Autobench spans, but the evidence model remains owned by Autobench. See [Native Instrumentation](native-instrumentation.md) for the typed fluent API, YAML DSL, compatibility doctor, privacy defaults, layered traces, and provider examples. ABP is the native collection protocol underneath these APIs. It owns signal ordering, task-local context, capture policy, instrumentation scope, trace materialization, and compatibility diagnostics. Instrumentors emit ABP evidence directly; they do not create OpenTelemetry spans and then convert them back into Autobench records. ## Manual Spans ```python from autobench import DurationMetricSpec, Semantic, SpanKind def run_case(ctx, case): with ctx.span( "support_agent", kind=SpanKind.AGENT, input=case.input, duration_metric=DurationMetricSpec( name="agent_latency", semantic_type=Semantic.TIME_LATENCY, unit="ms", ), ) as agent: result = call_agent(case.input) agent.set_output(result) agent.outcome(result["ok"]) return result ``` Span duration is calculated when the context manager closes. Nested spans preserve parent-child relationships and retain evidence emitted before an exception. ## Span Kinds `SpanKind` includes: - agent - LLM - tool - retriever - parser - workflow - custom Kinds are semantic selectors, not restrictions. A domain can use custom kinds and tags while generic agentic scorers continue selecting standard spans. ## Method Instrumentation `instrument_method` is the high-level helper for one class method. It records evidence only while a `RunContext` is active: ```python from autobench import InstrumentMetricSpec, Semantic, instrument_method handle = instrument_method( SearchClient, "search", span="search.request", metrics=[ InstrumentMetricSpec( name="result_count", semantic_type="retrieval.result_count", value_factory=lambda call: len(call.result), ), InstrumentMetricSpec( name="request_count", semantic_type="llm.requests", value_path="result.usage.requests", ), ], ) try: run_benchmark() finally: handle.close() ``` Instrumentation supports: - instance, static, class, and inherited methods; - synchronous and asynchronous calls; - iterators and generators, including `send`, `throw`, and early close; - asynchronous iterators and generators, including `asend`, `athrow`, and `aclose`; - synchronous and asynchronous context managers. The wrapper preserves the original descriptor, callable signature, return value, exception identity, and lazy streaming behavior. A stream span ends when the stream actually completes, fails, times out, or closes, so its duration is not merely the time required to construct an iterator. `value_factory` is the typed Python extraction seam. It receives an `InstrumentCall` containing the bound instance, arguments, result, error, stream item count, and last stream item. `value_path` is the declarative alternative for trusted attribute, mapping, and zero-argument accessor paths. Autobench does not execute arbitrary YAML expressions. Extraction and lifecycle callback errors are recorded as evidence or compatibility diagnostics. They do not replace the application's result or exception. The returned `InstrumentationHandle` is also a context manager and restores the original method on close. ## Scoped Suppression Instrumentation can be suppressed for the current task without changing global process state: ```python from autobench import suppress_instrumentation with suppress_instrumentation("search.client"): result = client.search("internal health check") ``` Suppression keys can identify an instrumentor or an operation family. Unrelated instrumentors stay active, nested scopes compose, and context tokens are reset even when application code raises. An empty `suppress_instrumentation()` scope suppresses all ABP instrumentation in the current task. ## Native Instrumentors Reusable SDK integrations implement the `Instrumentor` contract: ```python from autobench import ( AbstractionLayer, CaptureMechanism, Compatibility, InstrumentationHandle, InstrumentationRuntime, InstrumentorInfo, ) class ClientInstrumentor: info = InstrumentorInfo( id="example.client", version="1.0.0", target_distribution="example-client", supported_versions=">=2,<3", mechanism=CaptureMechanism.HOOK, layer=AbstractionLayer.CLIENT, span_kinds=("client.request",), semantic_families=("request", "response"), ) def check(self) -> Compatibility: return Compatibility.compatible() def install(self, runtime: InstrumentationRuntime) -> InstrumentationHandle: unsubscribe = register_native_callback(...) return InstrumentationHandle(unsubscribe, info=self.info) ``` Install instrumentors directly through one manager when building a custom integration: ```python from autobench import InstrumentationManager with InstrumentationManager() as manager: compatibility = manager.check(ClientInstrumentor()) if compatibility.installable: manager.install(ClientInstrumentor()) run_benchmark() ``` `InstrumentorInfo` declares stable identity, target package and version range, mechanism, layer, semantic families, source convention, optional dependencies, and sync/async/streaming/native-hook capabilities. `Compatibility` distinguishes compatible, degraded, unavailable, unsupported, and conflicting installations. Missing or incompatible optional dependencies degrade only the feature that needs them; a missing required target package prevents installation. Installing the same instrumentor version twice increments an owner reference count instead of installing duplicate hooks. Closing the final handle unregisters native callbacks or restores the exact patched descriptor. Competing owners can instrument the same method independently, while an external wrapper replacement produces a conflict diagnostic instead of being overwritten. Mechanisms should be selected in this order: 1. stable native processor or callback; 2. stable native wrapper/decorator extension point; 3. public method patch; 4. explicitly version-pinned private method patch; 5. unsupported with a compatibility diagnostic. Application benchmarks normally use the higher-level lifecycle owner instead: ```python from autobench import Benchmark, HTTPXInstrumentation, OpenAIInstrumentation benchmark = Benchmark("chat").instrument( OpenAIInstrumentation(), HTTPXInstrumentation(), ) result = benchmark.run() ``` `Benchmark.instrument(...)` installs configured and custom instrumentors before any matrix item, keeps them active through concurrent runs and streams, and closes them after execution. ## Trace Envelopes Adapters can normalize a completed external trace into `TraceEnvelope`: ```python from autobench import TraceEnvelope, attach_trace trace = TraceEnvelope( trace_id="trace-42", name="checkout-agent", input={"cart_id": "c1"}, output={"status": "complete"}, spans=tuple(converted_spans), attributes={"framework": "custom-agent-runtime"}, ) attach_trace(ctx, trace) ``` `attach_trace` preserves spans and errors and projects known usage, model, provider, duration, and outcome fields into semantic observations. Large native trace payloads should be written as an artifact and referenced by `raw_artifact`. ## Pydantic AI Usage Install the optional native instrumentor when the application uses Pydantic AI: ```bash pip install 'autobench[pydantic-ai]' ``` The integration uses Pydantic AI's public capability hooks and only injects its capability while an Autobench run is active: ```python from autobench import Benchmark, PydanticAIInstrumentation experiment = benchmark.instrument(PydanticAIInstrumentation()).run() ``` No manual span or metric calls are required. The instrumentor captures: - agent runs and streamed execution; - model requests, requested and response model identities, providers, and direct usage; - tool argument validation, execution, retry, failure, approval, and deferred control flow; - structured-output validation; - first-chunk latency, partial streams, failures, and normal completion; - tracked prompt, tool, and output-schema versions; - multimodal metadata, with binary references only when the capture policy requests full content. The instrumentor composes with user event handlers and Pydantic AI's own `Instrumentation` capability. It does not configure, replace, or require OpenTelemetry. Supported Autobench 0.1.x builds pin the public integration seam to Pydantic AI 2.22.x; `InstrumentationManager.check()` reports incompatible versions before installing hooks. Application outputs and exceptions are passed through unchanged. Aggregate agent usage and direct model usage retain distinct accounting scopes, and cost remains a downstream derivation. Replaying recorded ABP evidence does not require Pydantic AI to be installed. See the [live Pydantic AI example](examples.md#pydantic-ai) for a tool-using, structured-output, streaming benchmark with a retry path. ### Usage Bridge Pydantic AI usage can be normalized without importing Pydantic AI into core: ```python from autobench import PydanticAIUsage, record_pydantic_ai_usage record_pydantic_ai_usage( ctx, PydanticAIUsage( requests=1, input_tokens=420, output_tokens=83, model_name="gemini-3-flash-preview", provider="openrouter", ), ) ``` The bridge emits canonical LLM token, model, and provider observations that pricing derivation and reports can consume. ## Trace Extraction And Accounting Instrumentors record immutable facts. Extractors turn a completed ABP trace into semantic observations without mutating that trace: ```python from autobench import ( CompositeExtractor, SignalExtractor, SpanExtractor, UsageExtractor, replay_extraction, ) extractor = CompositeExtractor( SignalExtractor(), SpanExtractor(), UsageExtractor(), ) derived_record = replay_extraction(record, extractor) ``` The extractors have separate ownership: - `SignalExtractor` reconstructs measurements and events and preserves their accounting scope, abstraction layer, logical operation ID, and instrumentor identity. - `SpanExtractor` derives generic operation counts, direct durations, maximum depth and fan-out, critical-path makespan, parallelism, incomplete work, retry/recovery, validation, approval, tool-call, message-growth, and reference evidence. - `UsageExtractor` derives LLM request, token, requested-model, response-model, and provider evidence. It never derives cost. Every extractor has a stable name and version. Replay records both in extraction evidence and RunRecord lineage. Replaying a newer version replaces observations owned by the older version in the derived record; the parent record remains unchanged. ### Direct And Aggregate Evidence ABP keeps all raw measurements but prevents framework/client nesting from inflating totals: 1. Aggregate parent measurements are never added to direct child measurements. 2. Usage totals select one abstraction layer per semantic, preferring client evidence before framework, application, and transport evidence. 3. Equivalent direct operations with a shared logical operation ID are counted once. 4. Equal equivalent values are deduplicated. Conflicting values require a unique explicit authority; unresolved conflicts produce `ambiguous_direct_measurement` and are excluded from the derived total. 5. Aggregate values are retained as validation evidence. A disagreement with the direct total produces `aggregate_measurement_mismatch`. 6. Requested and response model identities remain separate factors. Reports and `ObservationQuery.first_exact()` prefer an accounting-safe aggregate summary over same-source per-operation direct evidence. Raw and projected queries can still inspect every underlying observation. Graph timing uses monotonic timestamps only. `time.critical_path` is the observed trace makespan, and `operation.parallelism` is completed leaf work divided by that makespan. Invalid or partial clock evidence is retained through diagnostics rather than repaired with wall-clock subtraction. ## Adapter Boundary Core instrumentation intentionally does not know Pydantic AI, OpenAI Agents, LangChain, DSPy, or OpenTelemetry internals. An integration should: 1. Collect from the framework's stable hooks. 2. Convert native calls or traces into Autobench spans and observations. 3. Store large raw payloads as artifacts. 4. Keep native dependencies optional. This boundary lets applications use existing instrumentation while RunRecords remain portable. --- ## Native Instrumentation Canonical page: https://vcoderun.github.io/autobench/native-instrumentation/ # Native Instrumentation Autobench native instrumentors collect ABP traces from supported SDKs without task-level `ctx.span()` or `ctx.metric()` calls. They are optional adapters around public hooks or pinned, reviewed patch points. Core benchmark, record, replay, and report imports do not require any of the instrumented SDKs. ## Install Install one integration or the complete set: ```bash pip install 'autobench[pydantic-ai]' pip install 'autobench[openai]' pip install 'autobench[openai-agents]' pip install 'autobench[httpx]' pip install 'autobench[instrumentation]' ``` The integration registry is lazy. Loading a YAML spec, replaying evidence, or running `autobench instrumentation doctor` does not import an SDK that is not installed. ## Automatic Discovery Use `instrument_all()` when the benchmark should activate every built-in integration that is installed and compatible in the current environment: ```python from autobench import Benchmark benchmark = Benchmark("support-agent").instrument_all() ``` Unavailable or unsupported integrations are skipped by default and recorded on each run as `instrumentation.skipped` diagnostic evidence. Use `strict=True` when the environment must support the complete selected set: ```python benchmark = Benchmark("support-agent").instrument_all( exclude={"httpx"}, strict=True, ) ``` Explicit settings take precedence over discovery, including an explicit `false`. A custom runtime instrumentor with the same instrumentor ID also takes precedence, so automatic discovery does not install a duplicate. Calling `instrument_all()` again replaces the previous automatic settings. ### Live OpenRouter Trace The live Pydantic AI example exercises automatic discovery across all three active layers: ```bash uv sync --extra instrumentation export OPENROUTER_API_KEY=... export OPENROUTER_MODEL=openrouter:openai/gpt-5.6-luna uv run python examples/pydantic_ai/openrouter_instrument_all.py \ --record /tmp/autobench-openrouter ``` The benchmark itself only opts in once: ```python benchmark = ( Benchmark("openrouter-shopping-agent") .instrument(PydanticAI(assets=[INSTRUCTIONS])) .instrument_all() ) ``` The explicit Pydantic AI instrumentor contributes tracked prompt metadata. Automatic discovery recognizes its instrumentor ID and does not install a duplicate, then adds the OpenAI client and HTTPX transport instrumentors. One real request therefore records agent, model, tool, output validation, stream, client request, and transport spans with their native parentage. It also records model identity, token usage, durations, HTTP method/host/path/status, score observations, asset versions, capture diagnostics, and replayable source provenance. HTTP bodies and credentials remain redacted by the default capture policy. The full source is `examples/pydantic_ai/openrouter_instrument_all.py`. It deliberately contains no manual `ctx.span()` or `ctx.metric()` calls so the resulting record demonstrates native collection rather than hand-authored benchmark telemetry. ## YAML Instrumentation belongs to the named benchmark: ```yaml # yaml-language-server: $schema=schemas/0.2.0/benchmark_schema.json benchmark: support-agent: dataset: source: file://datasets/cases.yaml run: python: support_benchmark:run variants: baseline: factors: model.name: openrouter:openai/gpt-5.6-luna instrumentation: all: exclude: [httpx] strict: false pydantic_ai: {} openai: {} httpx: capture: path: hash request_headers: [x-request-id] response_headers: [x-request-id] request_body: false response_body: false max_body_bytes: 65536 ``` Use `false` to retain a known integration in a shared spec without installing it: ```yaml instrumentation: openai_agents: false ``` Unknown integration names and unknown settings fail validation. YAML never evaluates Python expressions. The `all` block follows the same precedence rules as the Python builder. In this example HTTPX is excluded from discovery but its explicit capture settings still install it; all explicit entries remain authoritative. ## Python The fluent API accepts typed, serializable settings: ```python from autobench import ( Benchmark, HTTPXCaptureSettings, HTTPXInstrumentation, OpenAIInstrumentation, ) benchmark = Benchmark("streaming-chat").instrument( OpenAIInstrumentation(), HTTPXInstrumentation( capture=HTTPXCaptureSettings( path="hash", response_headers=("x-request-id",), ) ), ) experiment = benchmark.run() ``` It also accepts a custom `Instrumentor` instance. Runtime instances are installed for the whole benchmark matrix and closed even when execution fails. They are intentionally not serialized into the YAML spec: ```python benchmark.instrument(MyNativeInstrumentor(settings)) ``` Duplicate instrumentor IDs are rejected before hooks are installed. This avoids ambiguous ownership when a typed setting and a custom instance configure the same integration. ## Built-In Integrations | Integration | Layer | Collection seam | Evidence | | --- | --- | --- | --- | | Pydantic AI | framework | public agent capability | agent/model/tool/validation spans, messages, structured output, usage, stream lifecycle | | OpenAI Python | client | reviewed public client methods and stream types | chat, responses, embeddings, raw responses, model identity, direct usage, stream lifecycle | | OpenAI Agents | framework | native trace processor | workflow, agent, generation, tool, handoff, guardrail, and custom spans | | HTTPX | transport | public transport methods | request method/host/path policy, status, selected headers, body metadata, stream lifecycle | Run compatibility diagnostics before a benchmark: ```bash autobench instrumentation doctor ``` The Rich output shows availability, installed version, supported range, abstraction layer, mechanism, sync/async/streaming capabilities, span kinds, semantic families, capture defaults, and degradation diagnostics. ## Layered Traces Instrumentors compose instead of flattening one another. A Pydantic AI request using the OpenAI client over HTTPX can produce this parent chain: ```text task agent llm framework operation OpenAI client operation HTTP request ``` Transport spans do not emit token or cost usage. Framework aggregate usage and client direct usage retain different accounting scopes. Trace extraction selects one authoritative direct layer and keeps aggregate values as validation evidence, so enabling HTTPX cannot inflate LLM totals. ## Streaming Lifecycle A stream span does not end when an SDK returns an iterator. It remains open until the stream: - completes normally; - raises; - is cancelled; - is explicitly closed early; - is abandoned when the instrumentor manager closes. ABP records first-chunk evidence, item/chunk counts, partial state, and the final end reason. Native items, exceptions, iterator methods, and context-manager behavior pass through unchanged. ## HTTP Privacy Defaults HTTPX capture defaults are deliberately conservative: - query-free path hash, not the raw path; - no request or response headers unless named; - authorization, cookies, API keys, tokens, passwords, and secrets always redacted; - no request or response body capture; - bounded capture when bodies are explicitly enabled; - binary bodies represented by metadata and a digest, not embedded bytes. `path: full` is an explicit opt-in. Query strings and URL user information are not recorded by the path setting. Capture policies apply before evidence reaches a RunRecord. ## Trace Diagnostics Every native span records an `InstrumentationScope`: instrumentor and target versions, mechanism, abstraction layer, and source convention. Source facts can be retained alongside canonical Autobench semantic attributes. Unsupported library versions fail installation instead of silently patching an unknown lifecycle. Inspect recorded trace shape without importing task modules or optional SDKs: ```bash autobench instrumentation trace runs/support-agent/exp_... ``` The command reports per-case span/root counts, partial traces, diagnostics, span-kind totals, and instrumentor composition. ## Replay Without SDKs RunRecords contain materialized ABP traces, not live provider objects. A reporting or optimization worker can replay and re-extract evidence with only Autobench installed: ```python from autobench import CompositeExtractor, SignalExtractor, SpanExtractor, UsageExtractor from autobench.records.replay import load_run_record, replay_extraction record = load_run_record(path, root_dir=run_dir) derived = replay_extraction( record, CompositeExtractor(SignalExtractor(), SpanExtractor(), UsageExtractor()), ) ``` Extraction creates a derived record with lineage; it never mutates the original record. ## ABP And OpenTelemetry ABP is not an OpenTelemetry wrapper and has no OTel dependency. It is Autobench's evidence protocol for benchmark execution, semantic measurements, accounting scope, partial streams, replay, and optimization lineage. Native instrumentors use the same kinds of stable SDK hooks that mature OTel instrumentations validate, but emit ABP directly. A future bridge can export ABP spans to OTLP systems such as Logfire or Datadog. That bridge will be an adapter: ABP remains the source evidence model, and importing Autobench will not require an OTel SDK or collector. ## Protocol Stability ABP protocol version `1` is the initial public serialized contract. Autobench `0.2.x` preserves the meaning of its signal, trace, scope, provenance, and accounting fields. Readers retain unknown additive data through extension maps, while a breaking wire-format change requires a new protocol version. Instrumentor patch points are compatibility-gated separately because provider SDK lifecycles can change independently of ABP. ## Examples - `examples/abp_manual`: explicit workflow spans plus method instrumentation. - `examples/abp_concurrent`: concurrent sibling operations with task-local parentage. - `examples/pydantic_ai`: tool use, retry, streaming, and structured output; OpenAI models add OpenAI and HTTPX layers. - `examples/abp_openai`: offline official OpenAI streaming over an HTTPX mock transport. - `examples/abp_openai_agents`: offline native OpenAI Agents trace-processor workflow. - `examples/abp_replay`: trace extraction from recorded evidence without importing provider SDKs. --- ## Compatibility Contract Canonical page: https://vcoderun.github.io/autobench/abp-compatibility/ # ABP Compatibility Contract This page freezes the observable behavior preserved while the Autobench Instrumentation Protocol (ABP) replaces legacy span and instrumentation internals. Phases 1 through 8 now satisfy this contract. The complete public instrumentation guide is in [Instrumentation And Traces](instrumentation-and-traces.md). ## Compatibility Boundary The following top-level imports remain available while ABP is introduced: ```python from autobench import ( ArtifactRef, AssetVersion, DurationMetricSpec, ErrorRecord, InstrumentationHandle, InstrumentCall, InstrumentFactorSpec, InstrumentMetricSpec, Observation, RunContext, RunRecord, Span, SpanKind, SpanRecord, TraceEnvelope, attach_trace, get_active_run_context, instrument_method, trace_to_observations, ) ``` ABP may move implementations into new packages, but these imports and their current behavior remain compatibility facades until a separately announced deprecation cycle. ### Manual spans Existing manual spans preserve these guarantees: - `ctx.span(...)` is a synchronous context manager; - nested spans receive the active span as `parent_id`; - every completed span has UTC start/end timestamps and a non-negative monotonic duration; - a configured duration metric is linked to the span; - metrics, factors, events, errors, and artifacts retain their span link; - exceptions are recorded and then propagated; - `Span.set_output`, `Span.set_attribute`, and `Span.set_usage` continue to update the recorded span; - entering instrumentation without an active run context remains a no-op; - closing the final `InstrumentationHandle` restores the original descriptor. The tests in `tests/test_abp_compatibility.py` are the executable form of this contract. ### Stored evidence Legacy model-shaped RunRecord and TraceEnvelope YAML remains loadable. ABP will add protocol data additively and preserve the existing `spans` input during migration. Replay must not require the task module or an optional instrumented SDK. The frozen legacy examples are: - `tests/fixtures/abp/legacy_run_record.yaml` - `tests/fixtures/abp/legacy_trace_envelope.yaml` ## Concurrency Regression Contract `RunContext` now uses task-local ABP context. The concurrency migration is covered by passing regression tests for all of these cases: 1. concurrent sibling spans under one parent both point to that parent; 2. a nested task inherits the parent active at task creation; 3. completing one sibling does not change the other sibling's active parent; 4. out-of-order completion does not corrupt later parent selection; 5. cancellation closes only the cancelled branch and restores its context; 6. separate RunContexts never share active spans. The old mutable-stack reproduction is retained only in design history; it is not the current runtime behavior. ## Canonical Trace Decision ABP will have one canonical immutable `Trace` model. `TraceEnvelope` does not have behavior that justifies a second trace representation, so it will become a compatibility name for `Trace` rather than a parallel model. Existing `TraceEnvelope(...)`, `attach_trace(...)`, and `trace_to_observations(...)` callers continue to work. This avoids conversion drift between manually attached traces and traces materialized from native ABP signals. ## Package Shape ABP code is introduced only when its phase needs it. Empty placeholder modules are not created. ```text autobench/ protocol/ ids.py values.py signals.py traces.py context.py capture.py collector.py instrumentation/ models.py manager.py patching.py streaming.py pydantic_ai.py openai.py openai_agents.py httpx.py ``` Small modules are combined when separation would only create navigation cost. Existing unrelated modules are not moved as part of ABP. ## Optional Integration Targets The initial integration extras are reserved as follows: | Extra | Research baseline | First implementation phase | | --- | ---: | ---: | | `autobench[pydantic-ai]` | Pydantic AI 2.22.0 | 10 | | `autobench[openai]` | OpenAI Python 2.52.0 | 11 | | `autobench[openai-agents]` | OpenAI Agents 0.19.2 | 11 | | `autobench[httpx]` | HTTPX 0.28.1 | 12 | | `autobench[instrumentation]` | all integrations above | 13 | These versions are the public-API research baseline captured on 2026-08-03, not a compatibility claim. Dependency metadata is added only when each native instrumentor and its version matrix exist. Autobench core remains free of these dependencies. ## Manual Span Performance Baseline The baseline measures a minimal completed manual span with no observations, artifacts, or errors. Each repeat creates one RunContext and records 10,000 spans. Timing uses `timeit.repeat`; duration comes from the host monotonic clock. The benchmark does not enforce a CI latency threshold because shared CI timing is not stable. Reproduce it with: ```bash uv run python scripts/benchmark_spans.py --iterations 10000 --repeats 7 ``` Baseline captured before ABP runtime changes: | Field | Value | | --- | ---: | | Date | 2026-08-03 | | Python | 3.11.13 | | Platform | macOS 26.1 arm64 | | Minimum | 3,227.4 ns/span | | Median | 3,324.8 ns/span | The raw per-repeat values were `3467.0`, `3308.4`, `3299.3`, `3394.4`, `3394.3`, `3324.8`, and `3227.4` ns/span. Later phases compare using the same script and workload; they do not compare unrelated machine results. Release measurements captured after ABP materialization on the same host: | Workload | Measurement | | --- | ---: | | Manual ABP span | 28,836.7 ns/span median | | HTTPX baseline request | 35,346.2 ns/request median | | Instrumented HTTPX request | 224,263.6 ns/request median | | HTTPX instrumentation overhead | 188,917.4 ns/request median | | 10,000 x 32-byte HTTP stream | 2,618.0 ns/chunk median | | Long-stream peak allocation | 30,918 bytes | The long-stream result is about `3.1` peak allocated bytes per emitted chunk, which confirms the instrumentor does not retain chunk payloads as the stream grows. These numbers characterize this machine and are not release thresholds. Reproduce transport and stream measurements with: ```bash uv run python scripts/benchmark_spans.py --httpx --iterations 1000 --repeats 7 uv run python scripts/benchmark_spans.py --httpx-stream --chunks 10000 --chunk-size 32 --repeats 7 ``` --- ## YAML Spec Canonical page: https://vcoderun.github.io/autobench/yaml-spec/ # YAML Spec Autobench is YAML-first. Python builders compile to the same internal `BenchmarkSpec`. Every YAML file written by Autobench includes a `yaml-language-server` schema header that points to the versioned schema cache under `~/.autobench//schemas/`. ## Authoring Sections The authoring DSL places the benchmark ID under `benchmark` and keeps all behavior inside that named benchmark: | Section | Required | Purpose | | --- | --- | --- | | `description` | No | Human-readable benchmark intent | | `dataset` | Yes | Inline or file-backed cases, defaults, version, and metadata | | `run` | For execution | Python task target | | `variants` | Yes | Named factor combinations | | `score` | No | Built-in or Python scorers | | `derive` | No | Per-run semantic derivation such as token cost | | `post_derive` | No | Cross-run derivation such as paired baseline | | `policies` | No | Semantic metric constraints | | `report` | No | Leaderboard, matrix, comparisons, and distributions | | `semantic_registry` | No | Custom semantic definitions and aliases | ## Complete Authoring Example ```yaml # yaml-language-server: $schema=./schemas/0.2.0/benchmark_schema.json benchmark: support-routing: description: Compare current and candidate routing behavior. dataset: source: file://datasets/cases.yaml version: v2 defaults: tags: [regression] run: python: benchmark_tasks:run_case variants: baseline: factors: model: value: openrouter:openai/gpt-5.6-luna semantic: llm.model.name prompt_version: value: route-v3 semantic: prompt.version optimize: true candidate: factors: model: value: openrouter:openai/gpt-5.6-luna semantic: llm.model.name prompt_version: value: route-v4 semantic: prompt.version optimize: true score: route_correctness: exact: actual: output.route expected: case.expected.route semantic: quality.correctness goal: maximize role: objective success: pass: output.ok semantic: result.success role: constraint derive: - kind: token_cost pricing: file://pricing/models.yaml output: name: request_cost semantic_type: money.cost unit: usd direction: minimize role: constraint policies: - name: must-succeed metric: result.success must_equal: true report: leaderboard: show: accuracy: metric: quality.correctness aggregate: ratio_true total_cost: metric: money.cost aggregate: sum matrix: metric: quality.correctness compare: baseline -> candidate: show: accuracy: metric: quality.correctness aggregate: ratio_true ``` ## Resolution Rules - File references resolve relative to the benchmark YAML. - Python targets use `module:callable` and receive inferred search paths from the spec directory. - Duplicate case and variant IDs are validation errors. - A nonempty runnable matrix requires a task. - Scorer definitions must select exactly one scoring action. - Remote file references are rejected; price-source URL loading is an explicit integration API. - Custom semantics should be declared in the semantic registry. ## Shape ```yaml benchmark: support-routing: description: Deterministic support routing benchmark. dataset: source: file://datasets/cases.yaml defaults: metadata: owner: docs run: python: app.benchmarks.support:run_ticket_case variants: route_v1: factors: prompt_version: value: route-v1 semantic: prompt.version optimize: true routing_profile: baseline score: routing_correctness: exact: actual: output.queue expected: case.expected.queue semantic: quality.correctness tool_arguments: expected_action: metric: arguments observed_kind: tool span: kind: tool semantic: agent.tool.argument.correctness report: leaderboard: show: pass_rate: metric: result.success aggregate: ratio_true ``` ## Exported Benchmark YAML When Autobench renders a benchmark spec back to YAML, it uses a DSL-like shape instead of a raw model dump: ```yaml benchmark: support-routing: description: Route support tickets. dataset: source: datasets/cases.yaml cases: - id: ticket_1 input: subject: Refund run: python: app.benchmarks.support:run_ticket_case variants: route_v1: factors: prompt_version: value: route-v1 semantic: prompt.version optimize: true routing_profile: baseline score: success: pass: output.matched semantic: result.success goal: maximize report: leaderboard: show: pass_rate: metric: result.success aggregate: ratio_true ``` ## Notes - `dataset.source` supports local `file://` references and globs. - task targets use `module:function`. - variant factors accept either mapping or list form. - YAML does not execute inline expressions. - importable code hooks such as Python scorers remain explicit dotted targets. - `score..span` can target component spans by kind, name, tag, path, or semantic type. - `expected_action` scores compare `case.expected.actions` or `case.expected.tool_calls` with observed spans. ## Native Instrumentation The optional `instrumentation` section installs ABP SDK integrations for the complete benchmark matrix: ```yaml benchmark: support-agent: instrumentation: all: exclude: [httpx] strict: false pydantic_ai: {} openai: {} openai_agents: false httpx: capture: path: hash request_headers: [x-request-id] response_headers: [x-request-id] request_body: false response_body: false max_body_bytes: 65536 ``` `all` discovers every installed, compatible built-in integration. Missing integrations are skipped and recorded as run diagnostics unless `strict: true` is set. `exclude` accepts `pydantic_ai`, `openai`, `openai_agents`, and `httpx`. An explicit entry, including `false`, overrides discovery; the explicit HTTPX block above therefore remains enabled despite the discovery exclusion. `{}` selects privacy-safe defaults. `false` disables a known integration. Unknown integration names, settings, exclusions, or HTTP capture modes are validation errors. Optional SDKs are imported only when their enabled integration is resolved for execution. Replay never resolves this section. The versioned `benchmark_schema.json` describes this surface, so YAML language servers complete integration names and capture settings. See [Native Instrumentation](native-instrumentation.md) for the lifecycle and privacy contract. ## Safe Extensibility YAML is intended to be shareable and replayable. For that reason: - file references are resolved relative to the spec path - remote URLs are rejected - inline Python expressions are not part of the YAML surface ## Exported Run Record YAML Run records are the immutable per-case/per-variant evidence files used by replay. The trace signal objects below are abridged; recorded files retain their timestamps, sequence IDs, execution references, scope provenance, and captured attributes: ```yaml record: type: run version: 4 protocol: name: abp version: 1 semantic_registry: 1 run: id: run_ticket_1_route_v1 experiment: exp_support_routing_20260507T120000Z benchmark: support-routing case: ticket_1 variant: route_v1 status: passed outcome: evaluation: passed task: passed case: id: ticket_1 input: subject: Refund expected: queue: billing variant: id: route_v1 factors: prompt_version: value: route-v1 semantic: prompt.version optimize: true scores: routing_correctness: value: true semantic: quality.correctness role: objective metrics: measurements: routing_correctness: id: observation_1 name: routing_correctness kind: metric value: true semantic: quality.correctness diagnostics: latency_ms: value: 12.4 semantic: time.latency unit: ms trace: protocol: abp protocol_version: 1 trace_id: 70d8f4b6742d412a85cb7a198db07fe1 execution: benchmark_id: support-routing experiment_id: exp_support_routing_20260507T120000Z run_id: run_ticket_1_route_v1 case_id: ticket_1 variant_id: route_v1 root_span_ids: [3f2f6c57b9f56a11] spans: - span_id: 3f2f6c57b9f56a11 operation: benchmark.run kind: task scope: instrumentor_name: autobench.manual instrumentor_version: 0.2.0 package_name: autobench package_version: 0.2.0 mechanism: manual layer: application status: ok end_reason: completed measurements: [] events: [] links: [] references: [] partial: false links: [] references: [] diagnostics: [] signals: - type: span_start protocol: abp protocol_version: 1 span_id: 3f2f6c57b9f56a11 operation: benchmark.run kind: task - type: span_end protocol: abp protocol_version: 1 span_id: 3f2f6c57b9f56a11 status: ok reason: completed partial: false spans: call_router: kind: workflow started_at: "2026-05-07T12:00:00Z" duration: 0.0124 attributes: component: router lookup_user: kind: tool parent: call_router input: user_id: u1 output: tier: gold duration: 0.004 artifacts: generated_spec: media: application/x-yaml path: artifacts/run_ticket_1_route_v1/generated_spec.yaml assets: prompt.router: version: 7c91d4d7b1af output: queue: billing ``` When the serialized ABP trace exceeds the inline limit, the same section becomes a compact summary and artifact reference: ```yaml trace: id: 70d8f4b6742d412a85cb7a198db07fe1 partial: false spans: 7 signals: 31 artifact: id: abp_trace name: ABP trace media: application/vnd.autobench.abp-trace+yaml path: artifacts/run_ticket_1_route_v1/trace.yaml ``` ## Exported Dataset YAML Dataset exports use a DSL-like shape instead of raw model dumps: ```yaml record: type: dataset version: 1 dataset: id: tickets version: v1 metadata: owner: support defaults: tags: [smoke] cases: - id: ticket_1 input: subject: Refund ``` ## Exported Semantic Registry YAML Semantic registry exports use stable type ids with compact metadata: ```yaml record: type: semantic_registry version: 1 semantic_registry: version: 1 aliases: quality.answer: quality.score types: money.cost: unit: usd shape: number serving.cost: parent: money.cost unit: usd shape: number ``` ## Exported Pricing YAML Pricing tables are helper data, not a required runtime dependency. They keep provider/model aliases and tiered token prices readable: ```yaml record: type: pricing version: 1 pricing: provider: openrouter source: genai-prices updated_at: "2026-05-07" models: google/gemini-3-flash-preview: name: Gemini 3 Flash Preview aliases: - google:gemini-3-flash-preview - openrouter/google/gemini-3-flash-preview input: unit: mtok price: 0.3 tiers: - up_to: 1000000 price: 0.3 - price: 0.6 output: unit: mtok price: 2.5 cache_read: unit: mtok price: 0.03 ``` ## Exported Report YAML Report exports keep the summary under a single `report:` body: ```yaml record: type: report version: 1 report: benchmark: support-routing experiment: exp_support_routing_20260507T120000Z runs: 6 status: passed: 5 failed: 1 variants: baseline: factors: model.name: openrouter:openai/gpt-5.6-luna leaderboard: baseline: runs: 2 metrics: avg_coverage: 0.82 cases: ticket_1: baseline: status: passed metrics: coverage (coverage.ratio): 0.8 matrix: metric: coverage.ratio cases: ticket_1: baseline: 0.8 compare: baseline -> candidate: runs: 2 confounded: true distributions: cost_distribution: semantic: money.cost variants: baseline: [0.01, 0.02] ``` ## Exported Experiment YAML Experiment records keep replay data structured, but the outer shape stays readable: ```yaml record: type: experiment version: 4 experiment: id: exp_support_routing_20260507T120000Z benchmark: support-routing benchmark: id: support-routing dataset: id: tickets version: v1 hash: 9b5d... cases: - ticket_1 - ticket_2 counts: cases: 2 variants: 3 runs: 6 warnings: [] spec: hash: a13c... snapshot: benchmark: id: support-routing runs: count: 6 passed: 5 failed: 1 errored: 0 skipped: 0 paths: - cases/ticket_1/route_v1/run.yaml files: /abs/path/autobench.yaml: 3c4d... environment: python: "3.11.13" platform: macOS-15.5-arm64-arm-64bit cwd: /workspace/autobench semantic_registry: version: 1 aliases: quality.answer: quality.score types: money.cost: unit: usd shape: number ``` ## Exported Artifact YAML Artifacts are split into metadata and payload files. Text payloads stay as text. Structured payloads are wrapped so they remain recognizable YAML records: ```yaml record: type: artifact version: 1 artifact: id: trace name: trace media_type: application/x-yaml span_id: call_router payload: artifacts/run_ticket_1_route_v1/trace.yaml ``` ```yaml record: type: artifact_payload version: 1 artifact: id: trace name: trace media_type: application/x-yaml payload: steps: - tool: route_ticket arguments: queue: billing ``` ## Exported Asset YAML Tracked assets are stored as a readable index plus per-asset history files: ```yaml record: type: asset_index version: 1 assets: tool.create_car: kind: tool name: create_car semantic: agent.tool current_version: 7c91d4d7b1af file: tool_create_car.yaml ``` ```yaml record: type: asset version: 1 asset: id: tool.create_car kind: tool name: create_car semantic: agent.tool current_version: 7c91d4d7b1af doc: Create a new car instance. params: make: type: Literal["audi", "bmw", "mercedes"] required: true model: type: str required: true returns: type: Car asset_id: type.Car versions: - version: 15aa0dbceb02 state: kind: tool name: create_car params: make: type: Literal["audi", "bmw", "mercedes"] required: true hashes: content: ... changes: fields: [initial] - version: 7c91d4d7b1af parent: 15aa0dbceb02 state: kind: tool name: create_car params: make: type: Literal["audi", "bmw", "mercedes"] required: true hashes: content: ... source: ... source: path: ./vsh.py changes: fields: - params.year.type diff: | --- 15aa0dbceb02 +++ 7c91d4d7b1af @@ ... ``` --- ## Python API Canonical page: https://vcoderun.github.io/autobench/python-api/ # Python API Autobench exposes typed models and functions for every core layer. The `Benchmark` builder is a compact convenience API; direct `BenchmarkSpec` construction provides the complete configuration surface. ## Builder Example ```python from autobench import Benchmark, Case, ExactScorer, FactorValue, PassFailScorer, Semantic, Variant result = ( Benchmark("builder-demo") .dataset([Case(id="case_1", expected={"answer": "ok"})]) .variants( [ Variant(id="v1", factors=[FactorValue(name="enabled", value=True)]), {"id": "v2", "factors": {"enabled": False}}, ] ) .task("my_app.benchmarks:run_case") .scoring( [ PassFailScorer( name="success", path="output.success", semantic_type=Semantic.RESULT_SUCCESS, ), ExactScorer( name="answer", actual="output.answer", expected="case.expected.answer", semantic_type=Semantic.QUALITY_CORRECTNESS, ), ] ) .run() ) ``` Builder methods cover description, dataset, variants, task, scoring, per-run derivation, spec compilation, and sync/async execution. `to_spec()` returns the canonical `BenchmarkSpec`. For post-derivation, policies, report configuration, or a custom semantic registry, construct or update the typed spec before calling `run_benchmark_spec`: ```python from autobench import BenchmarkSpec, PolicySpec, run_benchmark_spec spec = BenchmarkSpec.model_validate(payload) spec = spec.model_copy( update={ "policies": [ PolicySpec( name="quality-gate", metric="quality.correctness", must_greater_equal=0.9, ) ] } ) result = await run_benchmark_spec(spec, concurrency_limit=4) ``` ## Task Signature Python task targets use: ```python def run_case(ctx, case): ... ``` `ctx` is always the first parameter. `case` is always the second. Tasks may be sync or async. ## Context Utilities `RunContext` and `Span` provide: - `metric` - `factor_observation` - `event` - `diagnostic` - `outcome` - `check` - `metrics` - `record_measurement` - `artifact` - `error` Span duration is owned by Autobench. Tasks do not need to hand-roll `perf_counter` timing for benchmark spans. ## Agentic Evidence Agent and workflow runs can record typed spans: ```python from autobench import Semantic, SpanKind def run_case(ctx, case): with ctx.span("support_agent", kind=SpanKind.AGENT) as agent: agent.metric("task_completed", True, semantic_type=Semantic.AGENT_TASK_COMPLETION) with ctx.span("lookup_user", kind=SpanKind.TOOL, input={"user_id": "u1"}) as tool: tool.set_output({"tier": "gold"}) ``` Expected tool/action checks can be expressed as scorers: ```python from autobench import ExpectedActionScorer, Semantic, SpanSelector scorer = ExpectedActionScorer( name="tool_arguments", semantic_type=Semantic.AGENT_TOOL_ARGUMENT_CORRECTNESS, metric="arguments", span=SpanSelector(kind="tool"), ) ``` Cases can use either `expected.actions` or `expected.tool_calls`: ```python Case( id="refund", expected={ "actions": [ {"tool": "lookup_user", "args": {"user_id": "u1"}, "order": 1}, ] }, ) ``` External framework traces can be attached with `TraceEnvelope`, and Pydantic AI usage can be recorded through `PydanticAIUsage` without making either OpenTelemetry or Pydantic AI a core dependency. ## Programmatic Layers | Layer | Primary APIs | | --- | --- | | Data | `Case`, `CaseDefaults`, `DatasetSpec`, `Variant`, `FactorValue` | | Spec | `BenchmarkInfo`, `BenchmarkSpec`, `TaskSpec`, `load_benchmark_spec`, `build_benchmark_plan` | | Runtime | `RunContext`, `Span`, `run_benchmark_spec`, `run_benchmark_path`, `expand_matrix` | | Native instrumentation | typed integration settings, `InstrumentationManager`, registry status, compatibility diagnostics | | Evidence | `Observation`, `ObservationQuery`, `SemanticRegistry`, projection helpers | | Scoring | built-in scorer models, `ScoringCall`, `ScoreRecord`, `SpanSelector` | | Derivation | token cost, pricing models, paired-baseline derivation, policies, measurement | | Tracking | `track`, `TrackingRegistry`, tracked asset models and YAML views | | Records | `record_experiment`, record loaders, `replay_experiment`, environment capture | | Reports | report models, builders, comparison, aggregation, rendering, and exporters | | Feedback | `build_feedback_records`, `build_optimization_feedback_input` | ## Loading And Running YAML ```python from pathlib import Path from autobench import load_benchmark_spec, run_benchmark_path spec = load_benchmark_spec(Path("autobench.yaml")) result = await run_benchmark_path( Path("autobench.yaml"), experiment_id="candidate-42", concurrency_limit=4, ) ``` `load_benchmark_spec` supports authoring DSL and normalized model shapes. It merges custom semantic registries with built-ins and resolves file-backed datasets and pricing relative to the spec. ## Recording And Replay ```python from pathlib import Path from autobench import record_experiment, replay_experiment record_experiment(result, Path("runs/candidate-42")) replayed = replay_experiment(Path("runs/candidate-42")) ``` Replay returns normal runtime result models but never imports the task target. ## Reports And Exports ```python from pathlib import Path from autobench import build_report, export_runs_csv, export_summary_yaml report = build_report(replayed) export_summary_yaml(replayed, Path("analysis/summary.yaml")) export_runs_csv(replayed, Path("analysis/runs.csv")) ``` Report builders can also be called independently: `build_leaderboard`, `build_case_matrix`, `compare_variants`, `build_metric_distribution`, and `build_run_metric_rows`. ## Typed Native Instrumentation Activate every compatible built-in integration available in the current environment: ```python from autobench import Benchmark benchmark = Benchmark("chat").instrument_all( exclude={"httpx"}, strict=False, ) ``` Discovery skips unavailable integrations and records why on each run. `strict=True` turns the first unavailable or unsupported selected integration into an `InstrumentationError`. Explicit typed settings and custom runtime instrumentors take precedence over their discovered equivalent. Configure individual integrations when capture settings must be controlled directly: ```python from autobench import ( Benchmark, HTTPXCaptureSettings, HTTPXInstrumentation, OpenAIInstrumentation, instrumentor_statuses, ) benchmark = Benchmark("chat").instrument( OpenAIInstrumentation(), HTTPXInstrumentation( capture=HTTPXCaptureSettings(path="hash", response_headers=("x-request-id",)) ), ) for status in instrumentor_statuses(): print(status.name, status.compatibility.status) ``` Settings are part of `BenchmarkSpec` and round-trip through the YAML DSL. Custom `Instrumentor` instances can use the same fluent method but remain runtime-only. See [Native Instrumentation](native-instrumentation.md). ## Extension Rules - Keep application execution in tasks. - Use custom Python scorers for domain evaluation, returning `ScoreRecord`. - Register domain semantics rather than overloading generic names. - Use adapters to convert external traces or usage into Autobench evidence. - Store large native payloads as artifacts. - Do not mutate recorded evidence; produce a new derived experiment or export. The complete signatures and model fields are available in [API Reference](api-reference.md). --- ## API Reference Canonical page: https://vcoderun.github.io/autobench/api-reference/ # API Reference This reference is generated from the installed public `autobench` package. The root package intentionally re-exports the supported API across data, specs, runtime, metrics, evaluation, tracking, records, reports, and errors. ::: autobench options: members: true members_order: source show_root_heading: true show_source: true show_signature_annotations: true separate_signature: true --- ## Development Canonical page: https://vcoderun.github.io/autobench/development/ # Development Autobench uses `uv` for dependency management and exposes stable repository operations through the Makefile. ## Environment ```bash uv sync --extra dev ``` The committed lock file is the reproducible dependency contract used by CI. ## Quality Gates ```bash make format make prod make pre-commit ``` `make prod` runs the test suite, enforces `100%` line and branch coverage, checks formatting and typing, builds the documentation, validates Python 3.11 through 3.13, and executes the offline examples end to end. ## Documentation The site uses Zensical's modern theme while retaining `mkdocs.yml` as the supported migration configuration format. ```bash make docs make docs-serve ``` Pushes to `main` build the site in strict mode. The workflow stores generated files in the `gh-pages` branch and deploys the same artifact through GitHub Pages Actions. ## Release Artifacts ```bash make build ``` The build produces a wheel and source distribution under `dist/`. Generated documentation, benchmark runs, internal planning files, references, and agent instructions are excluded from the published package. --- ## 0.2.0 Canonical page: https://vcoderun.github.io/autobench/release-notes/0.2.0/ # 0.2.0 Autobench `0.2.0` introduces the Autobench Instrumentation Protocol (ABP) and native evidence collection for supported AI and HTTP SDKs. ## Included - immutable ABP signals, traces, scopes, links, references, and protocol diagnostics - task-local trace context with correct concurrent parentage and partial-run preservation - privacy-first capture policy with redaction, truncation, hashing, and artifact references - versioned semantic source maps and accounting-safe trace extraction - native Pydantic AI, OpenAI Python, OpenAI Agents, and HTTPX instrumentors - sync, async, iterator, context-manager, and streaming lifecycle preservation - typed fluent and YAML instrumentation configuration with versioned schema completion - `autobench instrumentation doctor` compatibility diagnostics - `autobench instrumentation trace` replay-only trace summaries - real offline instrumentation, layering, and replay/extraction examples - Python 3.11, 3.12, 3.13, and 3.14 quality matrix - built-wheel/no-extras and target-library compatibility gates ## Compatibility Existing `0.1.0` benchmark specs and RunRecords remain loadable. ABP evidence is additive: manual spans and method instrumentation now materialize through the same protocol used by native instrumentors. Replaying ABP records does not import optional provider SDKs. ## Intentionally Deferred - OTLP and vendor exporters - distributed context propagation - import-hook auto-instrumentation - execution cassette replay - visualization ABP protocol version `1` is the initial public protocol. Autobench `0.2.x` will preserve its serialized meaning; additive fields remain forward-compatible through extension maps. A breaking wire-format change requires a new ABP protocol version. --- ## 0.1.0 Canonical page: https://vcoderun.github.io/autobench/release-notes/0.1.0/ # 0.1.0 Autobench `0.1.0` is the first release-shaped core. ## Included - YAML-first benchmark specs - deterministic task runtime - semantic observations and projection - scoring, derivation, post-derivation, and policies - immutable YAML recording and replay - Markdown, YAML, and CSV reporting - offline minimal, basic, mid, and advanced examples - real optional CodeMode dogfood integration - portable CLI source provenance - Python 3.11, 3.12, and 3.13 quality matrix ## Intentionally Not Included - autoptimize orchestration - GEPA integration - OpenTelemetry bridge - hosted dashboard features - distributed execution - full Pydantic Evals dataset/evaluator execution `PydanticEvalsBridge` in this release is an optional payload and availability bridge. It does not claim to execute Pydantic Evals datasets. The full internal evaluation runtime remains a later integration milestone.