Instrumentation And Traces¶

Autobench supports four collection styles that can be mixed in one run:

1. Explicit RunContext and Span calls inside a task.
2. Lightweight method instrumentation for existing application classes.
3. Trace-envelope adapters for an external agent or workflow runtime.
4. Native Pydantic AI, OpenAI, OpenAI Agents, and HTTPX instrumentors configured from Python or YAML.

OpenTelemetry is not a core dependency. Future OTLP bridges can export Autobench spans, but the evidence model remains owned by Autobench.

See Native Instrumentation for the typed fluent API, YAML DSL, compatibility doctor, privacy defaults, layered traces, and provider examples.

ABP is the native collection protocol underneath these APIs. It owns signal ordering, task-local context, capture policy, instrumentation scope, trace materialization, and compatibility diagnostics. Instrumentors emit ABP evidence directly; they do not create OpenTelemetry spans and then convert them back into Autobench records.

Manual Spans¶

from autobench import DurationMetricSpec, Semantic, SpanKind


def run_case(ctx, case):
    with ctx.span(
        "support_agent",
        kind=SpanKind.AGENT,
        input=case.input,
        duration_metric=DurationMetricSpec(
            name="agent_latency",
            semantic_type=Semantic.TIME_LATENCY,
            unit="ms",
        ),
    ) as agent:
        result = call_agent(case.input)
        agent.set_output(result)
        agent.outcome(result["ok"])
        return result

Span duration is calculated when the context manager closes. Nested spans preserve parent-child relationships and retain evidence emitted before an exception.

Span Kinds¶

SpanKind includes:

• agent
• LLM
• tool
• retriever
• parser
• workflow
• custom

Kinds are semantic selectors, not restrictions. A domain can use custom kinds and tags while generic agentic scorers continue selecting standard spans.

Method Instrumentation¶

instrument_method is the high-level helper for one class method. It records evidence only while a RunContext is active:

from autobench import InstrumentMetricSpec, Semantic, instrument_method

handle = instrument_method(
    SearchClient,
    "search",
    span="search.request",
    metrics=[
        InstrumentMetricSpec(
            name="result_count",
            semantic_type="retrieval.result_count",
            value_factory=lambda call: len(call.result),
        ),
        InstrumentMetricSpec(
            name="request_count",
            semantic_type="llm.requests",
            value_path="result.usage.requests",
        ),
    ],
)

try:
    run_benchmark()
finally:
    handle.close()

Instrumentation supports:

• instance, static, class, and inherited methods;
• synchronous and asynchronous calls;
• iterators and generators, including send, throw, and early close;
• asynchronous iterators and generators, including asend, athrow, and aclose;
• synchronous and asynchronous context managers.

The wrapper preserves the original descriptor, callable signature, return value, exception identity, and lazy streaming behavior. A stream span ends when the stream actually completes, fails, times out, or closes, so its duration is not merely the time required to construct an iterator.

value_factory is the typed Python extraction seam. It receives an InstrumentCall containing the bound instance, arguments, result, error, stream item count, and last stream item. value_path is the declarative alternative for trusted attribute, mapping, and zero-argument accessor paths. Autobench does not execute arbitrary YAML expressions.

Extraction and lifecycle callback errors are recorded as evidence or compatibility diagnostics. They do not replace the application's result or exception.

The returned InstrumentationHandle is also a context manager and restores the original method on close.

Scoped Suppression¶

Instrumentation can be suppressed for the current task without changing global process state:

from autobench import suppress_instrumentation

with suppress_instrumentation("search.client"):
    result = client.search("internal health check")

Suppression keys can identify an instrumentor or an operation family. Unrelated instrumentors stay active, nested scopes compose, and context tokens are reset even when application code raises. An empty suppress_instrumentation() scope suppresses all ABP instrumentation in the current task.

Native Instrumentors¶

Reusable SDK integrations implement the Instrumentor contract:

from autobench import (
    AbstractionLayer,
    CaptureMechanism,
    Compatibility,
    InstrumentationHandle,
    InstrumentationRuntime,
    InstrumentorInfo,
)


class ClientInstrumentor:
    info = InstrumentorInfo(
        id="example.client",
        version="1.0.0",
        target_distribution="example-client",
        supported_versions=">=2,<3",
        mechanism=CaptureMechanism.HOOK,
        layer=AbstractionLayer.CLIENT,
        span_kinds=("client.request",),
        semantic_families=("request", "response"),
    )

    def check(self) -> Compatibility:
        return Compatibility.compatible()

    def install(self, runtime: InstrumentationRuntime) -> InstrumentationHandle:
        unsubscribe = register_native_callback(...)
        return InstrumentationHandle(unsubscribe, info=self.info)

Install instrumentors directly through one manager when building a custom integration:

from autobench import InstrumentationManager

with InstrumentationManager() as manager:
    compatibility = manager.check(ClientInstrumentor())
    if compatibility.installable:
        manager.install(ClientInstrumentor())
        run_benchmark()

InstrumentorInfo declares stable identity, target package and version range, mechanism, layer, semantic families, source convention, optional dependencies, and sync/async/streaming/native-hook capabilities. Compatibility distinguishes compatible, degraded, unavailable, unsupported, and conflicting installations. Missing or incompatible optional dependencies degrade only the feature that needs them; a missing required target package prevents installation.

Installing the same instrumentor version twice increments an owner reference count instead of installing duplicate hooks. Closing the final handle unregisters native callbacks or restores the exact patched descriptor. Competing owners can instrument the same method independently, while an external wrapper replacement produces a conflict diagnostic instead of being overwritten.

Mechanisms should be selected in this order:

1. stable native processor or callback;
2. stable native wrapper/decorator extension point;
3. public method patch;
4. explicitly version-pinned private method patch;
5. unsupported with a compatibility diagnostic.

Application benchmarks normally use the higher-level lifecycle owner instead:

from autobench import Benchmark, HTTPXInstrumentation, OpenAIInstrumentation

benchmark = Benchmark("chat").instrument(
    OpenAIInstrumentation(),
    HTTPXInstrumentation(),
)
result = benchmark.run()

Benchmark.instrument(...) installs configured and custom instrumentors before any matrix item, keeps them active through concurrent runs and streams, and closes them after execution.

Trace Envelopes¶

Adapters can normalize a completed external trace into TraceEnvelope:

from autobench import TraceEnvelope, attach_trace

trace = TraceEnvelope(
    trace_id="trace-42",
    name="checkout-agent",
    input={"cart_id": "c1"},
    output={"status": "complete"},
    spans=tuple(converted_spans),
    attributes={"framework": "custom-agent-runtime"},
)

attach_trace(ctx, trace)

attach_trace preserves spans and errors and projects known usage, model, provider, duration, and outcome fields into semantic observations. Large native trace payloads should be written as an artifact and referenced by raw_artifact.

Pydantic AI Usage¶

Install the optional native instrumentor when the application uses Pydantic AI:

pip install 'autobench[pydantic-ai]'

The integration uses Pydantic AI's public capability hooks and only injects its capability while an Autobench run is active:

from autobench import Benchmark, PydanticAIInstrumentation

experiment = benchmark.instrument(PydanticAIInstrumentation()).run()

No manual span or metric calls are required. The instrumentor captures:

• agent runs and streamed execution;
• model requests, requested and response model identities, providers, and direct usage;
• tool argument validation, execution, retry, failure, approval, and deferred control flow;
• structured-output validation;
• first-chunk latency, partial streams, failures, and normal completion;
• tracked prompt, tool, and output-schema versions;
• multimodal metadata, with binary references only when the capture policy requests full content.

The instrumentor composes with user event handlers and Pydantic AI's own Instrumentation capability. It does not configure, replace, or require OpenTelemetry. Supported Autobench 0.1.x builds pin the public integration seam to Pydantic AI 2.22.x; InstrumentationManager.check() reports incompatible versions before installing hooks.

Application outputs and exceptions are passed through unchanged. Aggregate agent usage and direct model usage retain distinct accounting scopes, and cost remains a downstream derivation. Replaying recorded ABP evidence does not require Pydantic AI to be installed.

See the live Pydantic AI example for a tool-using, structured-output, streaming benchmark with a retry path.

Usage Bridge¶

Pydantic AI usage can be normalized without importing Pydantic AI into core:

from autobench import PydanticAIUsage, record_pydantic_ai_usage

record_pydantic_ai_usage(
    ctx,
    PydanticAIUsage(
        requests=1,
        input_tokens=420,
        output_tokens=83,
        model_name="gemini-3-flash-preview",
        provider="openrouter",
    ),
)

The bridge emits canonical LLM token, model, and provider observations that pricing derivation and reports can consume.

Trace Extraction And Accounting¶

Instrumentors record immutable facts. Extractors turn a completed ABP trace into semantic observations without mutating that trace:

from autobench import (
    CompositeExtractor,
    SignalExtractor,
    SpanExtractor,
    UsageExtractor,
    replay_extraction,
)

extractor = CompositeExtractor(
    SignalExtractor(),
    SpanExtractor(),
    UsageExtractor(),
)
derived_record = replay_extraction(record, extractor)

The extractors have separate ownership:

• SignalExtractor reconstructs measurements and events and preserves their accounting scope, abstraction layer, logical operation ID, and instrumentor identity.
• SpanExtractor derives generic operation counts, direct durations, maximum depth and fan-out, critical-path makespan, parallelism, incomplete work, retry/recovery, validation, approval, tool-call, message-growth, and reference evidence.
• UsageExtractor derives LLM request, token, requested-model, response-model, and provider evidence. It never derives cost.

Every extractor has a stable name and version. Replay records both in extraction evidence and RunRecord lineage. Replaying a newer version replaces observations owned by the older version in the derived record; the parent record remains unchanged.

Direct And Aggregate Evidence¶

ABP keeps all raw measurements but prevents framework/client nesting from inflating totals:

1. Aggregate parent measurements are never added to direct child measurements.
2. Usage totals select one abstraction layer per semantic, preferring client evidence before framework, application, and transport evidence.
3. Equivalent direct operations with a shared logical operation ID are counted once.
4. Equal equivalent values are deduplicated. Conflicting values require a unique explicit authority; unresolved conflicts produce ambiguous_direct_measurement and are excluded from the derived total.
5. Aggregate values are retained as validation evidence. A disagreement with the direct total produces aggregate_measurement_mismatch.
6. Requested and response model identities remain separate factors.

Reports and ObservationQuery.first_exact() prefer an accounting-safe aggregate summary over same-source per-operation direct evidence. Raw and projected queries can still inspect every underlying observation.

Graph timing uses monotonic timestamps only. time.critical_path is the observed trace makespan, and operation.parallelism is completed leaf work divided by that makespan. Invalid or partial clock evidence is retained through diagnostics rather than repaired with wall-clock subtraction.

Adapter Boundary¶

Core instrumentation intentionally does not know Pydantic AI, OpenAI Agents, LangChain, DSPy, or OpenTelemetry internals. An integration should:

1. Collect from the framework's stable hooks.
2. Convert native calls or traces into Autobench spans and observations.
3. Store large raw payloads as artifacts.
4. Keep native dependencies optional.

This boundary lets applications use existing instrumentation while RunRecords remain portable.