feat(harbor): add native Braintrust evaluation plugin

[Abhijeet Prasad (AbhiPrasad)]

Resolves SDK-64.

Overview

Adds a native Braintrust plugin for Harbor 0.20. Harbor continues to own task execution, containers, concurrency, retries, and verification; the plugin reconciles Harbor's authoritative final results into Braintrust datasets, experiments, scores, and traces.

This is intentionally not an OTLP adapter and does not call braintrust.Eval()—Harbor already owns the evaluation loop.

Harbor job
├── resolved task selection ───────────────► Braintrust Dataset
├── dataset + semantic agent variant ─────► Braintrust Experiment
└── retained final TrialResult ───────────► eval [root]
    ├── task
    │   ├── environment_setup
    │   ├── agent_setup
    │   ├── agent_execution
    │   │   └── ATIF task / llm / tool tree
    │   └── verification
    ├── normalized reward ────────────────► score child
    └── configured categorical outcome ──► classifier child

The detailed design is in docs/harbor-braintrust-plugin-design.md.

Setup and usage

Harbor requires Python 3.12+. The plugin ships in the normal Braintrust distribution and is discovered through the harbor.plugins entry-point group—users do not need to import or register it manually.

pip install 'harbor==0.20.0' braintrust

export BRAINTRUST_API_KEY=...
# Optional when the API key can access multiple organizations:
export BRAINTRUST_ORG_NAME='Braintrust SDKs'
# Optional plugin default; PROJECT_NAME is also accepted:
export HARBOR_BRAINTRUST_PROJECT=agent-benchmarks

Run it by selecting the braintrust plugin:

harbor run \
  -d terminal-bench/terminal-bench-2@latest \
  -a claude-code \
  -m anthropic/claude-sonnet-4-6 \
  -n 32 \
  --plugin braintrust

Plugin options can be supplied as Harbor kwargs:

harbor run ... \
  --plugin braintrust \
  --plugin-kwarg project_name=agent-benchmarks \
  --plugin-kwarg trajectory_mode=atif \
  --plugin-kwarg content_mode=messages \
  --plugin-kwarg 'score_keys=["correctness","pass_*"]' \
  --plugin-kwarg 'reward_rules={"error_rate":{"type":"score","direction":"minimize","min":0,"max":1}}'

Every constructor option has an equivalent HARBOR_BRAINTRUST_* environment fallback. Precedence is:

explicit constructor/--plugin-kwarg > HARBOR_BRAINTRUST_* > default

Standard BRAINTRUST_API_KEY, BRAINTRUST_ORG_NAME, and BRAINTRUST_APP_URL continue to configure the Braintrust connection.

Common configuration

Option / environment suffix	Values and behavior
project_name / PROJECT or PROJECT_NAME	Braintrust project name. Mutually exclusive with project_id.
dataset_mode / DATASET_MODE	sync (default) or none.
trajectory_mode / TRAJECTORY_MODE	atif (default), summary, or native.
content_mode / CONTENT_MODE	metadata, messages (default), or full.
attachments / ATTACHMENTS	none, verifier-details (default), or all.
artifact_include / ARTIFACT_INCLUDE	JSON glob array; requires attachments=all.
score_keys, metric_keys	JSON arrays of reward-key globs. Overlap is rejected before network I/O.
reward_rules	JSON object with exact per-reward score/metric and normalization rules.
classifier_rules	JSON object mapping classifier names to documented result paths.
invalid_score_policy	metric (default), drop, or error.
strict	Isolate plugin failures by default; set true to raise where Harbor permits.
max_content_bytes, max_attachment_bytes, max_total_attachment_bytes	Privacy/size bounds.

Public Python API

The public surface is intentionally small and Harbor-specific:

from braintrust.integrations.harbor import HarborPlugin, backfill_job

HarborPlugin

Harbor normally constructs this class through entry-point discovery. It can also be instantiated directly by code embedding Harbor:

plugin = HarborPlugin(
    project_name="agent-benchmarks",
    dataset_mode="sync",
    trajectory_mode="atif",
    content_mode="messages",
    score_keys=["correctness", "pass_*"],
    metric_keys=["runtime_*"],
    reward_rules={
        "error_rate": {
            "type": "score",
            "direction": "minimize",
            "min": 0,
            "max": 1,
            "score_name": "reliability",
        }
    },
    classifier_rules={"category": "agent_result.metadata.category"},
    strict=False,
)

It implements Harbor's asynchronous on_job_start(job) and on_job_end(job_result) protocol. Blocking Braintrust/filesystem work runs off Harbor's event loop.

backfill_job

Offline backfill uses the same identity, normalization, partitioning, reward, ATIF, and persistence core as online sync:

import asyncio
from braintrust.integrations.harbor import backfill_job

asyncio.run(
    backfill_job(
        "jobs/my-harbor-run",
        project_name="agent-benchmarks",
        trajectory_mode="atif",
    )
)

Backfill reads Harbor's persisted config, lock, job result, trial results, trajectories, verifier details, and artifact manifests. It does not rerun trials.

What appears in Braintrust

Datasets and records

Dataset sync is enabled by default.

• One managed dataset is created per Harbor source and exact logical task selection.
• Every resolved task becomes one record with a deterministic UUIDv5 ID.
• Record input contains task-authored semantics such as task identity, instruction, and multi-step instructions.
• expected remains null unless Harbor eventually provides a safe expected-output adapter; solution and verifier code are never treated as expected output.
• Metadata includes stable source identity, task digest/version, schema version, resource requirements, and normalized task metadata.
• A content change updates the dataset version; selecting a different task subset gets a different dataset scope.

Experiments and eval rows

A Harbor job is partitioned by:

(dataset identity, normalized semantic agent config, resolved skill digests)

Concurrency, retry policy, and output paths do not split experiments. Agent/model/kwargs, MCP configuration, resume behavior, safe environment configuration, and skill digests do.

Each retained final TrialResult becomes one root eval row. Harbor execution retries do not create extra experiment rows. Intentional n_attempts remain separate rows.

Root and canonical task spans share the same input, expected value, bounded output, or error. Output selection prefers standardized agent metadata, then the final non-copied ATIF agent message, per-step final messages, and finally a small status object.

Rewards, metrics, and classifications

Harbor rewards remain authoritative and the complete raw reward dictionary is retained at metadata.harbor.raw_rewards.

Reward classification is semantic rather than range-only:

1. Exact reward_rules entry.
2. score_keys / metric_keys glob.
3. Conventional reward is a score only when it is in [0, 1].
4. Invalid configured scores follow invalid_score_policy.
5. Every other numeric reward is an eval-root metric—even values that happen to be in [0, 1].

Normalized rewards create direct score children with purpose="scorer"; scores are not duplicated on the eval root. Explicit classifier rules create direct classifier children and grouped root classifications. One malformed classifier records a local warning/error without dropping the trial's numeric rewards.

Lifecycle and ATIF traces

Harbor's recorded timestamps shape the lifecycle tree:

eval
├── task
│   ├── environment_setup
│   ├── agent_setup
│   ├── agent_execution
│   │   ├── chat.completions.create [llm]
│   │   ├── calculator              [tool]
│   │   └── chat.completions.create [llm]
│   └── verification
├── correctness [score, scorer]
└── category    [classifier, scorer]

ATIF detail is conformance-gated:

• An llm leaf must represent exactly one model call and have provider/model identity, canonical messages, token usage, and valid timing.
• Tool leaves require arguments plus a correlated result or error, preserving call IDs and model → tool → model order.
• Deterministic, unknown, aggregated, redacted, or otherwise incomplete steps are safely downgraded to task summaries rather than mislabeled.
• Missing timestamps are interpolated within the agent phase; outliers are clamped and repairs are recorded on root metadata.
• Subagents become nested task trees.
• Usage maps to standard Braintrust token/cache/reasoning/cost metrics without aliases or fabricated zeroes.

Metadata, errors, and attachments

Every root includes a collision-safe metadata.harbor namespace with job/trial/task/agent/model identity, attempt/retry information, raw rewards, trajectory availability, custom metadata, reconciliation warnings, and artifact-manifest summaries.

• Harbor exceptions are logged on both root and canonical task; output is omitted.
• Missing reward without an exception is an unevaluated warning, not score zero.
• reward-details.json is bounded, normalized, redacted, and attached to scorer output by default.
• Allowlisted artifacts can be attached under agent_execution.output.artifacts with per-file and total limits.
• Absolute paths, secret-like fields, credentials, and oversized metadata are redacted or omitted.
• Braintrust credentials remain host-only and are never written into Harbor config, results, manifests, or traces.

Resume, retries, backfill, and failure isolation

Trial hooks feed reducer-based state machines, but authoritative rows are only dispatched from final JobResult.trial_results reconciliation. This prevents failed retry candidates from inflating experiment counts.

Deterministic dataset record IDs, experiment names, root IDs, and child span IDs make resume and backfill converge. Every job directory receives a credential-free braintrust-sync.json containing project/job identity, dataset and experiment IDs, trial terminal state, retry counts, warnings/errors, synced trial IDs, and completion state.

Default behavior isolates observability failures:

Failure	Behavior
Initialization/auth	Disable sync and persist a diagnostic manifest.
Hook event	Record the warning and continue.
Dataset sync	Continue with unassociated experiment rows where possible.
Malformed ATIF	Keep eval and lifecycle spans; omit/downgrade invalid detail.
Attachment	Omit it and record a warning.
Flush	Persist a local error for later backfill.

strict=True opts into raising unrecoverable failures where Harbor permits.

Packaging and tests

• Registers braintrust.integrations.harbor:HarborPlugin as the braintrust entry in harbor.plugins.
• Harbor remains optional; importing Braintrust without Harbor installed still works and Braintrust's Python minimum remains unchanged.
• Adds a pinned harbor==0.20.0 nox matrix session that skips interpreters below Python 3.12.
• Pure tests use real Harbor/Pydantic values for configuration, identity, rewards, classifications, metadata, and state reduction.
• The checked-in versioned VCR cassette exercises a real Harbor Trajectory through the real Braintrust SDK, uploads the emitted hierarchy, fetches it from Braintrust, and asserts persisted LLM/tool ordering, payloads, metrics, and span origin.
• Cassette request/response bodies redact absolute paths in addition to the repository's standard credential-header filtering.

cd py
nox -s 'test_harbor(latest)'
CI=1 nox -s 'test_harbor(latest)'  # strict cassette playback

Validation performed locally:

• test_harbor(latest): 9 passed
• test_core: 611 passed, 62 skipped, 12 xfailed
• Ruff, Pylint, pre-commit, stale-cassette, lockfile, and Python 3.10 import/compile checks passed