Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions .agents/rules/telemetry.md
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,7 @@ The deploy payload is declared twice: the proto in the universe repo (`proto/log
## Adding a new metric

1. Add the field to the relevant message in `libs/telemetry/protos/`, and populate it in `bundle/phases/telemetry.go`.
2. Refresh the acceptance golden: `go test ./acceptance -run '^TestAccept/bundle/telemetry' -update`. Anything engine-dependent belongs in a per-engine `out.*.$DATABRICKS_BUNDLE_ENGINE.txt` file rather than the engine-agnostic `out.telemetry.txt`; see the comments in that test's `script`.
2. Refresh the acceptance golden: `go test ./acceptance -run '^TestAccept/bundle/telemetry' -update`.
3. Send the universe PR adding the same field to the proto, with a `compliance.data_label` annotation matching the neighbouring fields.

**RULE: Never send a resource name, path, or any other user-authored string.** Names are PII. Resource identity goes through the numeric/UUID ID lists (`resource_job_ids` and friends), and only for types that have such an ID.
Expand All @@ -37,7 +37,7 @@ Other things worth knowing:
- A one-off boolean needs no proto change: add a key constant to `bundle/metrics/metrics.go` (snake_case, matching the constant name) and call `b.Metrics.SetBoolValue(...)`. It lands in `experimental.bool_values`.
- Metrics that may be removed later belong in `BundleDeployExperimental`, which carries no compatibility guarantees.
- Positional encodings such as the `upload_file_sizes` histogram are frozen once the metric has adoption: changing a bucket bound silently changes the meaning of every previously reported entry.
- Anything measured from the deployment state is engine-dependent. The direct engine keeps per-resource state in `resources.json` and reports sizes; the terraform path leaves `StateSizeBytes` zero and reports only counts.
- Per-resource state sizes come from the direct engine's state in `resources.json`.

## Where the data lands

Expand Down
4 changes: 2 additions & 2 deletions .agents/skills/bump-sdk/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -51,8 +51,8 @@ go test ./acceptance -run '^TestAccept$' -timeout=60m # MUST pass on its o
```

The verify pass is not optional.
Bundle tests run under an `EnvMatrix` of both engines (`terraform`, `direct`).
When a schema change makes one engine error while the other succeeds, the variants produce different output; `-update` runs both and each overwrites the other's `output.txt`, so it can silently settle on the passing variant and report `ok` while the golden is actually wrong.
Many bundle tests run under an `EnvMatrix` (e.g. `DMS`, `READPLAN`).
When a schema change makes one variant error while another succeeds, the variants produce different output; `-update` runs all of them and each overwrites the other's `output.txt`, so it can silently settle on the passing variant and report `ok` while the golden is actually wrong.
Only the non-update run catches this.
(Ignore `rejecting_proxy.go: blocking proxy` log lines, which are normal.
A test that times out under full parallel load but passes when run alone is a flake, not a regression.)
Expand Down
2 changes: 1 addition & 1 deletion .agents/skills/pr-checklist/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@ Before submitting a PR, run these commands to match what CI checks. CI uses the
# 2. Linting (CI runs full golangci-lint across all modules, not the diff-only wrapper)
./task lint

# 3. Tests (CI runs with both deployment engines)
# 3. Tests
./task test

# 4. If you changed bundle config structs, schema, or direct-engine resource code:
Expand Down
6 changes: 2 additions & 4 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -68,7 +68,7 @@ GIT_EDITOR=true GIT_SEQUENCE_EDITOR=true VISUAL=true GIT_PAGER=cat git rebase or
**bundle/** - Core bundle functionality for Declarative Automation Bundles
- `bundle/bundle.go` - Main Bundle struct and lifecycle management
- `bundle/config/` - Configuration loading, validation, and schema
- `bundle/deploy/` - Deployment logic (Terraform and direct modes)
- `bundle/deploy/` - Deployment logic (file upload, locking, state)
- `bundle/mutator/` - Configuration transformation pipeline
- `bundle/phases/` - High-level deployment phases

Expand All @@ -84,9 +84,7 @@ GIT_EDITOR=true GIT_SEQUENCE_EDITOR=true VISUAL=true GIT_PAGER=cat git rebase or

**Mutators**: Transform bundle configuration through a pipeline. Located in `bundle/config/mutator/` and `bundle/mutator/`. Each mutator implements the `Mutator` interface.

**Direct vs Terraform Deployment**: The CLI supports two deployment modes controlled by `DATABRICKS_BUNDLE_ENGINE` environment variable:
- `direct` (default) - Direct API calls without Terraform
- `terraform` - Uses Terraform for resource management
**Direct deployment engine**: Bundles are deployed by the direct engine (`bundle/direct/`), which calls the APIs without Terraform. The Terraform engine was removed in v1.20.0: `bundle.engine: terraform` and `DATABRICKS_BUNDLE_ENGINE=terraform` are rejected, and existing Terraform state is migrated to the direct engine before the command runs.

# Development Tips

Expand Down
2 changes: 1 addition & 1 deletion acceptance/bundle/help/bundle-deploy/output.txt
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@ Flags:
--force Force-override Git branch validation.
--force-lock Force acquisition of deployment lock.
-h, --help help for deploy
--plan string Path to a JSON plan file to apply instead of planning (direct engine only).
--plan string Path to a JSON plan file to apply instead of planning.
-q, --quiet count Reduce output: -q prints only the summary, -qq prints only warnings and errors.
--select strings Deploy only the specified resource (e.g. 'my_job' or 'jobs.my_job'). Can be repeated or comma-separated.

Expand Down
12 changes: 5 additions & 7 deletions bundle/bundle.go
Original file line number Diff line number Diff line change
Expand Up @@ -86,7 +86,7 @@ type Metrics struct {
// ResourceState is the direct engine's per-resource deployment state
// captured right after the deploy. It carries each resource's state-size in
// bytes so deploy telemetry can be derived without re-reading or re-parsing
// the state file. Nil for terraform deploys.
// the state file.
ResourceState resourcestate.ExportedResourcesMap
}

Expand Down Expand Up @@ -188,12 +188,10 @@ type Bundle struct {
// When non-empty, only the specified resources are included in deployment.
Select []string

// MigratingToDirect is set when the direct engine is requested but the existing
// state still uses terraform, so the state is migrated to the direct engine after
// this deploy. Resources that only the direct engine supports are skipped by this
// run rather than rejected: terraform cannot deploy them, and since terraform
// could never have deployed them they are absent from its state. The next deploy,
// which runs on the migrated state, creates them.
// MigratingToDirect is set when the existing state still uses terraform, so it is
// migrated to the direct engine in memory before the command runs (statemgmt.Migrate).
// Deploy and destroy commit the migration (statemgmt.CommitMigration); other commands
// leave the terraform state untouched.
MigratingToDirect bool

// Quiet is the output verbosity reduction requested via -q/--quiet, which is
Expand Down
4 changes: 2 additions & 2 deletions bundle/config/bundle.go
Original file line number Diff line number Diff line change
Expand Up @@ -46,8 +46,8 @@ type Bundle struct {
// Overrides the cluster used for jobs and other supported assets.
ClusterId string `json:"cluster_id,omitempty"`

// Engine specifies the deployment engine to use ("terraform" or "direct").
// Defaults to "direct"; set "terraform" to opt out.
// Engine specifies the deployment engine to use. "direct" is the only supported
// engine and the default; "terraform" is rejected with the removal error.
// Can be overridden with the DATABRICKS_BUNDLE_ENGINE environment variable.
Engine engine.EngineType `json:"engine,omitempty"`

Expand Down
2 changes: 1 addition & 1 deletion bundle/config/engine/engine.go
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@ const (
)

// Default is used when the user has not set the value, both for new bundles and
// for existing terraform deployments (which are migrated to it after a deploy).
// for existing terraform deployments (whose state is migrated to it before the command runs).
const Default = EngineDirect

// TerraformRemovedMessage is the error text shown when a bundle pins the removed
Expand Down
2 changes: 1 addition & 1 deletion bundle/config/experimental.go
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ type Experimental struct {
// single immutable snapshot rather than being synced individually. When true,
// the deployment calls /api/2.0/repos/snapshots with a zip of all files and sets
// workspace.file_path and workspace.artifact_path to the returned content-addressed
// path. Only supported with the direct deployment engine.
// path.
ImmutableFolder bool `json:"immutable_folder,omitempty"`

// By default Python wheel tasks deployed as is to Databricks platform.
Expand Down
14 changes: 6 additions & 8 deletions bundle/config/mutator/drop_empty_strings.go
Original file line number Diff line number Diff line change
Expand Up @@ -17,10 +17,10 @@ type dropEmptyStrings struct{}
// either literally (policy_id: "") or via a variable that resolved to "", so
// this must run after variable resolution.
//
// Both engines convert the resolved config through convert.ToTyped, which
// The direct engine converts the resolved config through convert.ToTyped, which
// force-sends explicitly-set zero values, defeating the omitempty tag. Dropping
// here fixes it uniformly for terraform and direct and makes the result visible
// in `bundle validate -o json`, which serializes the dynamic value.
// here fixes it and makes the result visible in `bundle validate -o json`, which
// serializes the dynamic value.
func DropEmptyStrings() bundle.Mutator {
return &dropEmptyStrings{}
}
Expand All @@ -43,11 +43,9 @@ func (m *dropEmptyStrings) Apply(ctx context.Context, b *bundle.Bundle) diag.Dia
return root, err
}

// It seems safe to send an empty description "" to the backend, but for apps
// it causes drift in terraform: the Apps API always returns "" for an unset
// description, so bundle/deploy/terraform/tfdyn/convert_app.go injects it
// anyway. To avoid an unnecessary difference between the direct and terraform
// engines, keep the empty apps description here instead of dropping it.
// Keep an empty apps description instead of dropping it: an app update without
// the field leaves the remote description unchanged, so removing description
// from config would never converge. Force-sending "" clears it.
return dyn.MapByPattern(root, dyn.NewPattern(dyn.Key("resources"), dyn.Key("apps"), dyn.AnyKey()), func(_ dyn.Path, app dyn.Value) (dyn.Value, error) {
if _, err := dyn.Get(app, "description"); err != nil {
return dyn.Set(app, "description", dyn.V(""))
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -25,9 +25,8 @@ type applyDefaultTaskSource struct{}
// Python code added or removed, and so the injected value is not exposed to Python
// code. That is the same rationale that first moved this defaulting out of the
// pre-Python resource mutators, at the cost of making it invisible in
// `bundle validate`/`summary` and of leaving the direct engine without it
// (terraform got it from tfdyn). Running it here as a normal mutator restores that
// visibility and covers both engines from one place.
// `bundle validate`/`summary` and of leaving the direct engine without it.
// Running it here as a normal mutator restores that visibility.
// See https://github.com/databricks/cli/pull/3359 and
// https://github.com/databricks/cli/pull/3528.
func ApplyDefaultTaskSource() bundle.Mutator {
Expand All @@ -39,7 +38,6 @@ func (a *applyDefaultTaskSource) Name() string {
}

// sourceAwareTaskTypes are the task types that support the `source` field.
// Keep in sync with supportedTypeTasks in bundle/deploy/terraform/tfdyn/convert_job.go.
// https://docs.databricks.com/api/workspace/jobs/create
var sourceAwareTaskTypes = []string{
"dbt_task",
Expand Down
2 changes: 0 additions & 2 deletions bundle/config/resources/model_serving_endpoint.go
Original file line number Diff line number Diff line change
Expand Up @@ -15,8 +15,6 @@ type ModelServingEndpoint struct {
BaseResource
ID string `json:"id,omitempty" bundle:"readonly"`

// This represents the input args for terraform, and will get converted
// to a HCL representation for CRUD
serving.CreateServingEndpoint

// This is a resource agnostic implementation of permissions for ACLs.
Expand Down
2 changes: 0 additions & 2 deletions bundle/config/resources/registered_model.go
Original file line number Diff line number Diff line change
Expand Up @@ -15,8 +15,6 @@ type RegisteredModel struct {
BaseResource
ID string `json:"id,omitempty" bundle:"readonly"`

// This represents the input args for terraform, and will get converted
// to a HCL representation for CRUD
catalog.CreateRegisteredModelRequest

// List of grants to apply on this registered model.
Expand Down
3 changes: 1 addition & 2 deletions bundle/direct/bundle_plan.go
Original file line number Diff line number Diff line change
Expand Up @@ -1218,8 +1218,7 @@ func (b *DeploymentBundle) makePlan(ctx context.Context, configRoot *config.Root
}

// Unescape "$${...}" to a literal "${...}" in the typed state, which is what
// gets deployed and saved. The terraform engine leaves the escape in place for
// Terraform itself to unescape; the direct engine has to do it here.
// gets deployed and saved.
//
// This runs on the typed state rather than the dynamic config so that
// extractReferences below still sees the escaped form and does not mistake
Expand Down
2 changes: 1 addition & 1 deletion bundle/direct/dresources/adapter.go
Original file line number Diff line number Diff line change
Expand Up @@ -79,7 +79,7 @@ type IResource interface {

// [Optional] IsEmptyState reports that newState describes no resource at all: the planner
// omits the node instead of planning a create, and apply drops the state entry instead of
// persisting one, so both engines converge on "this node does not exist".
// persisting one, so plan and apply converge on "this node does not exist".
// Example: func (*ResourceGrants) IsEmptyState(state *GrantsState) bool
IsEmptyState(newState any) bool

Expand Down
4 changes: 2 additions & 2 deletions bundle/internal/schema/annotations.yml
Original file line number Diff line number Diff line change
Expand Up @@ -622,7 +622,7 @@ resources:
"$fields":
"libraries":
"description": |-
A list of libraries to install on the cluster. Installed via the Libraries API after the cluster is created. Only supported in direct deployment mode.
A list of libraries to install on the cluster. Installed via the Libraries API after the cluster is created.
"lifecycle":
"description": |-
Settings that control the deployment lifecycle of the resource, such as preventing it from being destroyed.
Expand All @@ -632,7 +632,7 @@ resources:
Lifecycle setting to prevent the resource from being destroyed.
"started":
"description": |-
Lifecycle setting to deploy the resource in started mode. Only supported for apps, clusters, and sql_warehouses in direct deployment mode.
Lifecycle setting to deploy the resource in started mode. Only supported for apps, clusters, and sql_warehouses.
"permissions":
"description": |-
The permissions to apply to this resource.
Expand Down
10 changes: 5 additions & 5 deletions bundle/phases/telemetry.go
Original file line number Diff line number Diff line change
Expand Up @@ -300,11 +300,11 @@ func LogDeployTelemetry(ctx context.Context, b *bundle.Bundle, errMsg string) {
b.Metrics.SetBoolValue(metrics.AiRuntimeTaskMultitask, airMultitask)
}

// Record whether the deprecated terraform engine was explicitly opted into,
// separately per source. Only emitted when true; absence means "not opted in
// via this source". An invalid env value has already failed the command
// upstream via ResolveEngineSetting, so treating the FromEnv error as
// "not terraform" here cannot mask a real terraform opt-in.
// Record whether the removed terraform engine was still requested (the command
// then fails with the removal error), separately per source. Only emitted when
// true; absence means "not requested via this source". An invalid env value has
// already failed the command upstream via ResolveEngineSetting, so treating the
// FromEnv error as "not terraform" here cannot mask a real terraform request.
if b.Config.Bundle.Engine == engine.EngineTerraform {
b.Metrics.SetBoolValue(metrics.EngineTerraformConfig, true)
}
Expand Down
4 changes: 2 additions & 2 deletions bundle/schema/jsonschema.json

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

5 changes: 2 additions & 3 deletions bundle/statemgmt/direct_migration.go
Original file line number Diff line number Diff line change
Expand Up @@ -276,9 +276,8 @@ func convertTFStateToDirect(ctx context.Context, b *bundle.Bundle, tfState *migr
return tempStatePath, false, nil, errors.New("failed to apply secret scope fixups")
}

// b.Config has been modified by terraform.Interpolate which converts bundle-style
// references (${resources.pipelines.x.id}) to terraform-style (${databricks_pipeline.x.id}).
// BuildStateFromTF expects ${resources.*} references, so reverse the interpolation first.
// The config may use terraform-style references (${databricks_pipeline.x.id}).
// BuildStateFromTF expects ${resources.*} references, so rewrite them first.
uninterpolatedRoot, err := reverseInterpolate(b.Config.Value())
if err != nil {
return tempStatePath, false, nil, fmt.Errorf("failed to reverse interpolation: %w", err)
Expand Down
3 changes: 1 addition & 2 deletions bundle/statemgmt/reverse_interpolate.go
Original file line number Diff line number Diff line change
Expand Up @@ -8,8 +8,7 @@ import (
"github.com/databricks/cli/libs/dyn/dynvar"
)

// reverseInterpolate reverses the terraform.Interpolate transformation.
// It converts terraform-style resource references back to bundle-style.
// reverseInterpolate converts terraform-style resource references to bundle-style.
// Example: ${databricks_pipeline.my_etl.id} → ${resources.pipelines.my_etl.id}
func reverseInterpolate(root dyn.Value) (dyn.Value, error) {
return dynvar.Resolve(root, func(path dyn.Path) (dyn.Value, error) {
Expand Down
2 changes: 1 addition & 1 deletion cmd/apps/deploy_bundle.go
Original file line number Diff line number Diff line change
Expand Up @@ -78,7 +78,7 @@ func BundleDeployOverrideWithWrapper(wrapError ErrorWrapper) func(*cobra.Command
deployCmd.Flags().BoolVar(&opts.autoApprove, "auto-approve", false, "Skip interactive approvals that might be required for deployment.")
deployCmd.Flags().MarkDeprecated("compute-id", "use --cluster-id instead")
deployCmd.Flags().BoolVar(&opts.verbose, "verbose", false, "Enable verbose output.")
deployCmd.Flags().StringVar(&opts.readPlanPath, "plan", "", "Path to a JSON plan file to apply instead of planning (direct engine only).")
deployCmd.Flags().StringVar(&opts.readPlanPath, "plan", "", "Path to a JSON plan file to apply instead of planning.")
// Verbose flag currently only affects file sync output, it's used by the vscode extension
deployCmd.Flags().MarkHidden("verbose")
deployCmd.Flags().BoolVar(&opts.skipValidation, "skip-validation", false, "Skip project validation (build, typecheck, lint)")
Expand Down
4 changes: 2 additions & 2 deletions cmd/apps/import.go
Original file line number Diff line number Diff line change
Expand Up @@ -52,8 +52,8 @@ func newImportCommand() *cobra.Command {
Long: `(Experimental) Import an existing Databricks app and convert it to a bundle configuration.

This command creates a new bundle directory with the app configuration, downloads
the app source code, binds the bundle to the existing app, and deploys it using
direct deployment mode. This allows you to manage the app as code going forward.
the app source code, binds the bundle to the existing app, and deploys it. This
allows you to manage the app as code going forward.

The command will:
1. Create an empty bundle folder with databricks.yml
Expand Down
2 changes: 1 addition & 1 deletion cmd/bundle/deploy.go
Original file line number Diff line number Diff line change
Expand Up @@ -42,7 +42,7 @@ See https://docs.databricks.com/en/dev-tools/bundles/index.html for more informa
cmd.Flags().MarkDeprecated("compute-id", "use --cluster-id instead")
cmd.Flags().BoolVar(&verbose, "verbose", false, "Enable verbose output.")
cmd.Flags().CountVarP(&quiet, "quiet", "q", "Reduce output: -q prints only the summary, -qq prints only warnings and errors.")
cmd.Flags().StringVar(&readPlanPath, "plan", "", "Path to a JSON plan file to apply instead of planning (direct engine only).")
cmd.Flags().StringVar(&readPlanPath, "plan", "", "Path to a JSON plan file to apply instead of planning.")
cmd.Flags().StringSliceVar(&selectResources, "select", nil, "Deploy only the specified resource (e.g. 'my_job' or 'jobs.my_job'). Can be repeated or comma-separated.")
// Verbose flag currently only affects file sync output, it's used by the vscode extension
cmd.Flags().MarkHidden("verbose")
Expand Down
Loading
Loading