Skip to content

Asset Tracking

Benchmarks need to know which prompt, tool, schema, or configuration produced each result. Autobench tracking assigns content-derived versions, captures structured metadata, persists history, and binds exact asset versions to RunRecords.

Supported SDK instrumentors can discover these assets without decorators. Use explicit tracking on this page when the application owns a better logical identity or when the component never crosses an instrumented boundary. See Automatic Asset Discovery for unannotated Pydantic AI, OpenAI, OpenAI Agents, capability scopes, privacy, and custom SDK extraction.

Prompts And Text Assets

Track inline text:

from autobench import track

SYSTEM_PROMPT = track.prompt(
    name="support_system_prompt",
    text="Route the request to billing, account, or technical support.",
)

Or load it from a file:

SYSTEM_PROMPT = track.prompt(
    name="support_system_prompt",
    source="prompts/support.md",
)

TrackedPrompt.raw returns the text, and str(SYSTEM_PROMPT) provides the same value for APIs that expect a string. File-backed prompts retain their source path and source hash.

Tools

@track.tool preserves the callable's exact signature and return type while collecting tool metadata:

from typing import Literal

from autobench import track


@track.tool
def route_ticket(
    queue: Literal["billing", "account", "technical"],
    priority: int = 1,
) -> bool:
    """Route a ticket to a support queue."""
    return priority > 0

The resulting ToolAsset records:

  • qualified name and docstring
  • parameter names, kinds, annotations, defaults, and requirements
  • return annotation
  • source path and source hash
  • structured parameter schema
  • semantic type and version lineage

Annotations are normalized by structure rather than alias spelling. If the contents of a Literal, union, generic, model, or referenced type change, the asset hash changes even when the alias name stays the same.

Pydantic Models, Dataclasses, And Classes

from dataclasses import dataclass
from typing import Literal

from autobench import track
from pydantic import BaseModel, Field


@track.type
class Car(BaseModel):
    make: Literal["audi", "bmw", "mercedes"]
    model: str = Field(examples=["a3", "320i"])
    year: int = Field(gt=0)


@track.dataclass(frozen=True, slots=True)
class CarRequest:
    make: Literal["audi", "bmw", "mercedes"]
    model: str
    year: int

Pydantic models are hashed from normalized JSON Schema plus source identity. Standard dataclasses use dataclass field definitions and resolved annotations. Other typed classes use resolved class annotations, inspectable signatures, and source hashes.

TypeAsset and FieldAsset preserve field names, resolved annotations, descriptions, examples, aliases, defaults, required state, and relevant constraints.

Composing Another Class Decorator

When @track.type above a class-transforming decorator gives poor type-checker inference, use track.decorate_type:

from dataclasses import dataclass

from autobench import track


@track.decorate_type(dataclass, frozen=True, slots=True)
class Request:
    value: str

The decorator and its normalized arguments are stored as asset metadata. track.dataclass(...) is the typed convenience form for the standard dataclass decorator.

Arbitrary Assets

Use track.asset for configurations, policies, routing tables, or other application components:

@track.asset(kind="routing_policy", name="enterprise_routing")
def route_policy(ticket):
    return "priority" if ticket["enterprise"] else "standard"

The decorator returns the original object unchanged. Callables use source and signature metadata; manual version, hash, source_path, parent_version, and metadata values are available when automatic identity is not enough.

Versions, Diffs, And Persistence

TrackingRegistry keeps current assets and version history in memory during execution. Persist it with:

from pathlib import Path

from autobench import track

track.write_assets(Path(".autobench/assets"))

The directory contains an index, one lightweight manifest per asset, and content.sqlite3, a local content-addressed registry keyed by asset ID and version. Every new manifest version links to its parent and stores changed paths plus typed content and diff references. Prompt text, tool bodies, schema definitions, and readable diffs never appear in the manifests. SQLite transactions avoid rewriting the complete history as it grows, while content hashes deduplicate identical snapshots.

from autobench import load_asset_content

version = track.asset_version_of(SYSTEM_PROMPT)
snapshot = load_asset_content(
    Path(".autobench/assets/content.sqlite3"),
    asset_id=version.asset_id,
    version=version.version,
)
assert snapshot["raw"] == SYSTEM_PROMPT.raw

For a version with a parent, resolve its readable diff with load_asset_diff(path, asset_id=..., version=..., parent_version=...).

Recording persists every version a run used, not only the latest one, so an asset that changes several times inside one run keeps each snapshot and diff. Each version links to the latest version registered before it unless its producer declares the lineage itself, with at most one of:

  • parent_version: the version it derives from, for example when a revised prompt derives from an earlier version rather than the most recent one.
  • parent_fingerprint: the 64-character content fingerprint of the parent. Autobench derives the parent's version from it, so a producer that knows only the content it started from, such as an optimizer or a hand-written variant of a source prompt, can still name its parent.
  • root=True: the version derives from no earlier version, for example a source text that was edited by hand rather than produced from the previous one.

Diffs are computed against the declared parent. An older version recorded after a newer one is added to the history without moving current_version.

Binding Assets To Runs

def run_case(ctx, case):
    ctx.attach_tracked_asset(SYSTEM_PROMPT)
    ctx.attach_tracked_asset(route_ticket)
    ctx.attach_tracked_asset(Car)
    return execute(case.input)

The exact AssetVersion values are copied into the RunRecord. Reports and optimization feedback can then relate metric changes to prompt, tool, or output-schema versions without guessing from source control state. An asset trajectory compares each version with its parent on the cases both ran, and an asset comparison tests two versions, such as the root and the leaf of the lineage, against each other.