Skip to content

feat: Go eval support - #305

Open
David Elner (delner) wants to merge 1 commit into
mainfrom
feature/eval_go
Open

feat: Go eval support#305
David Elner (delner) wants to merge 1 commit into
mainfrom
feature/eval_go

Conversation

@delner

Copy link
Copy Markdown

What

Adds Go to EvalLanguage, so bt eval can run Go evals.

Why

The Braintrust Go SDK now implements the runner protocol (evalrunner package),
but bt could not dispatch to it: EvalLanguage was a closed
{JavaScript, Python} enum and detect_eval_language hard-errored on any other
extension. This is the companion change.

How

Go needs a shape bt did not have. For JS and Python, bt supplies the
program
— its own embedded eval-runner.{ts,py} — and the user supplies files
that program imports at runtime. Go has no runtime loading: the user's
compiled package is the runner
. So bt simply
executes what the user built. Go's compilation unit is also a directory, not a
file, so all inputs must resolve to one package.

bt eval --language go ./cmd/evals            # compile and run the package
bt eval ./cmd/evals                          # inferred from a *_eval.go marker
bt eval --language go --runner ./bin/evals   # run a prebuilt binary
bt eval --dev ./cmd/evals                    # dev server for the playground
  • Discovery: **/*_eval.go, a suffix mirroring Go's own _test.go
    convention. The matched file is only a marker — go run compiles the whole
    package. Pass --language go for a package with no such file.
  • Command: go run <package dir>. With --runner, the binary is spawned
    with no arguments — the Go runner takes all input from BT_EVAL_* and
    ignores argv, and appending files would break a main that parses flags.
  • Toolchain: BT_EVAL_GO_BIN / BT_EVAL_GO, then GOROOT, then PATH.

Everything else — env vars, the SSE socket, the stdout manifest, drive_eval_runner
is language-agnostic and unchanged.

Other behavior changed

Worth calling out, since these are not Go-specific:

  1. prepare_eval_runners is now lazy. It was writing both
    eval-runner.ts and eval-runner.py to disk on every spawn, before bt knew
    which language it was running — and under --dev that is once per HTTP
    request. It now runs inside the language arms that need it.
  2. vendor added to DEFAULT_EXCLUDE_DIRS, so bare bt eval in a Go repo
    does not pick up a vendored dependency's eval files. This also applies to
    JS/Python discovery, but only in the zero-argument case.
  3. **/*_eval.go added to DEFAULT_EVAL_GLOBS. A polyglot repo containing
    both TS evals and a *_eval.go file will now get "Mixed eval file types are
    not supported yet" from bare bt eval — an existing failure class, new
    trigger.
  4. expand_eval_file_globs now takes the --language override, because it
    runs before detect_eval_language and would otherwise reject a Go package
    directory before the override was ever consulted.

Watch mode

A Go binary cannot report its own sources, so the watch set is derived
statically: the package directory (whose mtime changes when files are added or
removed) plus its buildable .go files. Imports in other packages are not
followed.

Tests

17 new tests. detect_eval_language had no coverage at all before this, so Go
adds the first. Verified end to end against the Go SDK: /list, a streaming
playground run, and a CLI bt eval run.

@delner

Copy link
Copy Markdown
Author

Just of note: this includes support for both source and precompiled Go runners. If we prefer to simplify to one path, we could do that instead.

@github-actions

Copy link
Copy Markdown
Contributor

Latest downloadable build artifacts for this PR commit 7cdbb8da7e01:

Available artifact names
  • artifacts-build-global
  • artifacts-build-local-aarch64-pc-windows-msvc
  • artifacts-build-local-x86_64-pc-windows-msvc
  • artifacts-build-local-x86_64-apple-darwin
  • artifacts-build-local-x86_64-unknown-linux-musl
  • artifacts-build-local-aarch64-apple-darwin
  • artifacts-build-local-x86_64-unknown-linux-gnu
  • artifacts-build-local-aarch64-unknown-linux-gnu
  • artifacts-plan-dist-manifest
  • cargo-dist-cache

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant