feat(appkit): agent eval framework, judge, mlflow connector (stack 2/5) by MarioCadenas · Pull Request #478 · databricks/appkit

MarioCadenas · 2026-07-16T14:26:29Z

Stack 2/5 · targets pr/agent-evals-1-tracing (review after #1).

The core eval framework, plus LLM-as-judge and the MLflow REST connector.

Authoring (defineEval): drive an agent over HTTP against a running app; assert with t.succeeded(), t.calledTool(), t.check(value, matcher) (includes/equals/matches). Gate-by-default, .soft() to demote.
Native MLflow Evaluation runs: a run tagged genai_evaluate; each turn's trace links via mlflow.sourceRun; per-assertion feedback written via the assessments REST API.
LLM-as-judge (t.judge.factuality/closedQA/custom) via autoevals → a Databricks serving endpoint.
connectors/mlflow: MlflowClient (host/token, post/postResult, serving URL) + resolveDatabricksAuth/resolveWorkspaceClient (OAuth from a CLI profile — no hand-set PAT). Extracted so both evals and future callers share the REST/auth layer.
appkit agent eval CLI.

Squashed history note: contains the framework, judge, and connector-extraction commits.

github-actions · 2026-08-18T13:00:15Z

📦 Bundle size report

Compared against bundle-size-baseline.json (main).

`@databricks/appkit` ⚠️ over budget

npm tarball (packed): 945 KB (+58 KB) — gzipped download (dist + bin; excludes release-only docs/NOTICE).

dist	raw	gzip
JS (runtime)	965 KB (+60 KB)	338 KB (+21 KB)
Type declarations	365 KB (+21 KB)	129 KB (+9.6 KB)
Source maps	1.9 MB (+116 KB)	634 KB (+38 KB)
Other	11 KB	3.7 KB
Total	3.2 MB (+197 KB)	1.1 MB (+69 KB)

Per-entry composition (own code — deps external (as shipped))

Entry	Initial (gz)	Lazy (gz)	Total (gz)	node_modules (min)	Own code (min)
`.`	93 KB (+4.9 KB)	2.5 KB	96 KB (+4.9 KB)	external	306 KB (+17 KB)
`./beta`	59 KB (+9.0 KB)	457 B	60 KB (+9.0 KB)	external	177 KB (+29 KB)
`./testing`	17 KB	0 B	17 KB	external	50 KB
`./tsdown`	520 B	0 B	520 B	external	813 B
`./type-generator`	21 KB	0 B	21 KB	external	61 KB

Chunks:

Entry	Chunk	Load	Size (gz)
`.`	`index.js`	initial	89 KB
`.`	`utils.js`	initial	4.0 KB
`.`	`remote-tunnel-manager.js`	lazy	2.5 KB
`./beta`	`beta.js`	initial	43 KB
`./beta`	`stream-manager.js`	initial	5.8 KB
`./beta`	`wide-event-emitter.js`	initial	3.2 KB
`./beta`	`databricks.js`	initial	3.0 KB
`./beta`	`configuration.js`	initial	2.1 KB
`./beta`	`service-context.js`	initial	1.3 KB
`./beta`	`client.js`	initial	434 B
`./beta`	`client-options.js`	initial	220 B
`./beta`	`supervisor-api.js`	lazy	192 B
`./beta`	`databricks.js`	lazy	142 B
`./beta`	`index.js`	lazy	123 B
`./testing`	`index.js`	initial	17 KB
`./tsdown`	`index.js`	initial	520 B
`./type-generator`	`index.js`	initial	21 KB

`@databricks/appkit-ui`

npm tarball (packed): 348 KB (+5 B) — gzipped download (dist + bin; excludes release-only docs/NOTICE).

dist	raw	gzip
JS (runtime)	394 KB	132 KB
Type declarations	228 KB (+32 B)	83 KB (+8 B)
Source maps	764 KB	252 KB
CSS	16 KB	3.2 KB
Total	1.4 MB (+32 B)	471 KB (+8 B)

Per-entry composition (consumer bundle — deps bundled, peerDeps external)

Entry	Initial (gz)	Lazy (gz)	Total (gz)	node_modules (min)	Own code (min)
`./js`	5.3 KB	49 KB	55 KB	208 KB	14 KB
`./js/beta`	20 B	0 B	20 B	0 B	0 B
`./react`	432 KB	49 KB	481 KB	1.3 MB	177 KB
`./react/beta`	1.0 KB	0 B	1.0 KB	0 B	1.9 KB

Chunks:

Entry	Chunk	Load	Size (gz)
`./js`	`index.js`	initial	5.2 KB
`./js`	`chunk`	initial	120 B
`./js`	`apache-arrow`	lazy	49 KB
`./js/beta`	`beta.js`	initial	20 B
`./react`	`index.js`	initial	430 KB
`./react`	`tslib`	initial	2.1 KB
`./react`	`apache-arrow`	lazy	49 KB
`./react/beta`	`beta.js`	initial	1.0 KB

⚠️ Over budget: a package's shipped tarball, or a browser entry's consumer bundle (deps included), grew by more than 5% (and >10 KB). This check will fail — reduce the size, or acknowledge the increase by updating bundle-size-baseline.json.

github-actions · 2026-08-18T13:05:05Z

🤖 AppKit PR bot

🔬 Run evals

Start an eval for this PR from the evals-monitor app: Go to Evals Monitor →

📦 Try this PR's app template

Scaffolds a new app from this PR's SDK build. Run it in any folder (requires the GitHub CLI — gh auth login — and the Databricks CLI):

gh run download 32473494214 -R databricks/appkit -n appkit-template-0.64.0-pr.710c4d7-pr-agent-evals-2-framework-478 -D appkit-pr-478 \
  && unzip -o "appkit-pr-478/appkit-template-0.64.0-pr.710c4d7-pr-agent-evals-2-framework-478.zip" -d "appkit-pr-478" \
  && databricks apps init --template "appkit-pr-478"

The template pins @databricks/appkit and @databricks/appkit-ui to tarballs built from this branch, so the scaffolded app runs against this PR's code.

…runs eve-style eval authoring (defineEval + t-context + matchers) discovered from config/agents/<id>/evals/*.eval.ts and run via 'appkit agent eval' against a running app. Streams per-eval progress and gates CI via exit code. When Databricks creds + an experiment are set, it creates a real MLflow evaluation run (mlflow.runType=genai_evaluate): each eval's trace links to the run, pass/fail is written as feedback assessments, and aggregate metrics are logged. All via the MLflow REST API. Signed-off-by: MarioCadenas <MarioCadenas@users.noreply.github.com>

Extend the agent eval framework and tighten MLflow output to match the native `mlflow.genai.evaluate` experience: - LLM-as-judge via autoevals (factuality, closedQA, custom), pointed at a Databricks serving endpoint; exposed through `t.judge.*`. - One Feedback assessment per assertion (judges as LLM_JUDGE with score + rationale) plus an overall `appkit_eval`; assessment names sanitized to `[A-Za-z0-9_-]` since the API rejects dots. - Trace-table parity: set Request/Response previews and the `mlflow.traceName` tag (the Trace-name column reads the tag, not the span name). - Eval runs carry `mlflow.source.name`/`type` tags so linked traces show Source and Run name; live chat traces have no run so those stay empty. - Example judge eval under config/agents/query/evals. Signed-off-by: MarioCadenas <MarioCadenas@users.noreply.github.com>

Introduce connectors/mlflow as the shared REST + auth layer for MLflow, so the eval runner (and future callers) stop threading host/token and hand-rolling fetch/URL logic: - MlflowClient owns {host, token}: normalizes the host once, exposes post() (throws) for runs/* and postResult() (structured failure) for best-effort assessment writes, plus servingEndpointsUrl() for the judge. - resolveDatabricksAuth() mints an OAuth bearer from a CLI profile via the SDK WorkspaceClient (the AppKit-native path), so `agent eval` no longer requires a hand-set DATABRICKS_TOKEN. Adds an `--profile` flag. - Eval run create/finish, assessment reporting, and the judge take the client; the agents plugin's host normalization now delegates to the connector's normalizeHost. The mlflow-tracing SDK wrapper stays in the agents plugin: it manages a process-global provider (like TelemetryManager) and has an agent-shaped API, so it isn't a connector. Signed-off-by: MarioCadenas <MarioCadenas@users.noreply.github.com>

Signed-off-by: MarioCadenas <MarioCadenas@users.noreply.github.com>

Post-rebase integration with main's biome->oxc migration (#538) and the SDK-facade boundary rule (#534): - Route the mlflow connector's auth through createWorkspaceClient instead of importing @databricks/sdk-experimental directly (oxlint no-restricted-imports); behaviour is unchanged. - Apply oxfmt import grouping to the evals + connector files authored before the migration. Signed-off-by: MarioCadenas <MarioCadenas@users.noreply.github.com>

Remove dead surface flagged in review: - EvalConfig + defineEvalConfig: nothing loads evals.config.ts (the runner never reads it), so the type and helper configured nothing. - EvalDefinition.tags / .timeoutMs: never read by the runner. - discover: replace hand-rolled statSync/recursion with readdirSync({ recursive, withFileTypes }). - resolveEvalDefault: drop the seen-Set, redundant with the loop bound. Signed-off-by: MarioCadenas <MarioCadenas@users.noreply.github.com>

…er-agent layout Adapts the eval framework to #533's unified agent discovery: agents now live in server/agents/<id>/agent.{md,ts} (config/agents is a deprecated fallback), so evals move next to them. - discover.ts scans server/agents/<id>/evals/, reusing CODE_AGENTS_SOURCE_DIR and agentDirNames so eval discovery follows the same folder-selection policy (symlinked agent folders included) as the agents plugin. - Relocate the dev-playground example evals config/agents/query/evals -> server/agents/query/evals. - Update discover.test fixtures + doc strings. Signed-off-by: MarioCadenas <MarioCadenas@users.noreply.github.com>

Regenerated index + sidebar pick up main's defineManifest (#485) alongside the eval-framework entries. Signed-off-by: MarioCadenas <MarioCadenas@users.noreply.github.com>

MarioCadenas requested a review from a team as a code owner July 16, 2026 14:26

MarioCadenas requested review from pkosiec and removed request for a team July 16, 2026 14:26

MarioCadenas force-pushed the pr/agent-evals-1-tracing branch from 412b08d to f517306 Compare August 5, 2026 15:22

Base automatically changed from pr/agent-evals-1-tracing to main August 13, 2026 09:27

MarioCadenas force-pushed the pr/agent-evals-2-framework branch from 0b55719 to 1431ce7 Compare August 18, 2026 12:57

MarioCadenas force-pushed the pr/agent-evals-2-framework branch from 941dae8 to e32afdd Compare August 21, 2026 10:28

MarioCadenas added 8 commits August 21, 2026 12:31

docs(appkit): regenerate API reference after rebase onto main

bf8be0d

Signed-off-by: MarioCadenas <MarioCadenas@users.noreply.github.com>

docs(appkit): reconcile API reference after rebase onto main

65de478

Regenerated index + sidebar pick up main's defineManifest (#485) alongside the eval-framework entries. Signed-off-by: MarioCadenas <MarioCadenas@users.noreply.github.com>

MarioCadenas force-pushed the pr/agent-evals-2-framework branch from e32afdd to 65de478 Compare August 21, 2026 10:37

Navigation Menu

Uh oh!

feat(appkit): agent eval framework, judge, mlflow connector (stack 2/5) - #478

feat(appkit): agent eval framework, judge, mlflow connector (stack 2/5)#478
MarioCadenas wants to merge 8 commits into
mainfrom
pr/agent-evals-2-framework

MarioCadenas commented Jul 16, 2026

Uh oh!

github-actions Bot commented Aug 18, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Aug 18, 2026 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Conversation

MarioCadenas commented Jul 16, 2026

Uh oh!

github-actions Bot commented Aug 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

📦 Bundle size report

@databricks/appkit ⚠️ over budget

@databricks/appkit-ui

Uh oh!

github-actions Bot commented Aug 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🤖 AppKit PR bot

🔬 Run evals

📦 Try this PR's app template

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

github-actions Bot commented Aug 18, 2026 •

edited

Loading

`@databricks/appkit` ⚠️ over budget

`@databricks/appkit-ui`

github-actions Bot commented Aug 18, 2026 •

edited

Loading