feat: Go eval support - #305
Open
David Elner (delner) wants to merge 1 commit into
Open
Conversation
David Elner (delner)
force-pushed
the
feature/eval_go
branch
from
August 13, 2026 23:16
7dc21d0 to
7cdbb8d
Compare
Author
|
Just of note: this includes support for both source and precompiled Go runners. If we prefer to simplify to one path, we could do that instead. |
Contributor
|
Latest downloadable build artifacts for this PR commit
Available artifact names
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Adds
GotoEvalLanguage, sobt evalcan run Go evals.Why
The Braintrust Go SDK now implements the runner protocol (
evalrunnerpackage),but
btcould not dispatch to it:EvalLanguagewas a closed{JavaScript, Python}enum anddetect_eval_languagehard-errored on any otherextension. This is the companion change.
How
Go needs a shape
btdid not have. For JS and Python, bt supplies theprogram — its own embedded
eval-runner.{ts,py}— and the user supplies filesthat program imports at runtime. Go has no runtime loading: the user's
compiled package is the runner. So bt simply
executes what the user built. Go's compilation unit is also a directory, not a
file, so all inputs must resolve to one package.
**/*_eval.go, a suffix mirroring Go's own_test.goconvention. The matched file is only a marker —
go runcompiles the wholepackage. Pass
--language gofor a package with no such file.go run <package dir>. With--runner, the binary is spawnedwith no arguments — the Go runner takes all input from
BT_EVAL_*andignores argv, and appending files would break a
mainthat parses flags.BT_EVAL_GO_BIN/BT_EVAL_GO, thenGOROOT, thenPATH.Everything else — env vars, the SSE socket, the stdout manifest,
drive_eval_runner—is language-agnostic and unchanged.
Other behavior changed
Worth calling out, since these are not Go-specific:
prepare_eval_runnersis now lazy. It was writing botheval-runner.tsandeval-runner.pyto disk on every spawn, before bt knewwhich language it was running — and under
--devthat is once per HTTPrequest. It now runs inside the language arms that need it.
vendoradded toDEFAULT_EXCLUDE_DIRS, so barebt evalin a Go repodoes not pick up a vendored dependency's eval files. This also applies to
JS/Python discovery, but only in the zero-argument case.
**/*_eval.goadded toDEFAULT_EVAL_GLOBS. A polyglot repo containingboth TS evals and a
*_eval.gofile will now get "Mixed eval file types arenot supported yet" from bare
bt eval— an existing failure class, newtrigger.
expand_eval_file_globsnow takes the--languageoverride, because itruns before
detect_eval_languageand would otherwise reject a Go packagedirectory before the override was ever consulted.
Watch mode
A Go binary cannot report its own sources, so the watch set is derived
statically: the package directory (whose mtime changes when files are added or
removed) plus its buildable
.gofiles. Imports in other packages are notfollowed.
Tests
17 new tests.
detect_eval_languagehad no coverage at all before this, so Goadds the first. Verified end to end against the Go SDK:
/list, a streamingplayground run, and a CLI
bt evalrun.