diff --git a/.agents/rules/telemetry.md b/.agents/rules/telemetry.md index a4c6e20f41d..5526dfb2007 100644 --- a/.agents/rules/telemetry.md +++ b/.agents/rules/telemetry.md @@ -25,7 +25,7 @@ The deploy payload is declared twice: the proto in the universe repo (`proto/log ## Adding a new metric 1. Add the field to the relevant message in `libs/telemetry/protos/`, and populate it in `bundle/phases/telemetry.go`. -2. Refresh the acceptance golden: `go test ./acceptance -run '^TestAccept/bundle/telemetry' -update`. Anything engine-dependent belongs in a per-engine `out.*.$DATABRICKS_BUNDLE_ENGINE.txt` file rather than the engine-agnostic `out.telemetry.txt`; see the comments in that test's `script`. +2. Refresh the acceptance golden: `go test ./acceptance -run '^TestAccept/bundle/telemetry' -update`. 3. Send the universe PR adding the same field to the proto, with a `compliance.data_label` annotation matching the neighbouring fields. **RULE: Never send a resource name, path, or any other user-authored string.** Names are PII. Resource identity goes through the numeric/UUID ID lists (`resource_job_ids` and friends), and only for types that have such an ID. @@ -37,7 +37,7 @@ Other things worth knowing: - A one-off boolean needs no proto change: add a key constant to `bundle/metrics/metrics.go` (snake_case, matching the constant name) and call `b.Metrics.SetBoolValue(...)`. It lands in `experimental.bool_values`. - Metrics that may be removed later belong in `BundleDeployExperimental`, which carries no compatibility guarantees. - Positional encodings such as the `upload_file_sizes` histogram are frozen once the metric has adoption: changing a bucket bound silently changes the meaning of every previously reported entry. -- Anything measured from the deployment state is engine-dependent. The direct engine keeps per-resource state in `resources.json` and reports sizes; the terraform path leaves `StateSizeBytes` zero and reports only counts. +- Per-resource state sizes come from the direct engine's state in `resources.json`. ## Where the data lands diff --git a/.agents/skills/bump-sdk/SKILL.md b/.agents/skills/bump-sdk/SKILL.md index 8bdbe93c0b7..98f5d5179db 100644 --- a/.agents/skills/bump-sdk/SKILL.md +++ b/.agents/skills/bump-sdk/SKILL.md @@ -51,8 +51,8 @@ go test ./acceptance -run '^TestAccept$' -timeout=60m # MUST pass on its o ``` The verify pass is not optional. -Bundle tests run under an `EnvMatrix` of both engines (`terraform`, `direct`). -When a schema change makes one engine error while the other succeeds, the variants produce different output; `-update` runs both and each overwrites the other's `output.txt`, so it can silently settle on the passing variant and report `ok` while the golden is actually wrong. +Many bundle tests run under an `EnvMatrix` (e.g. `DMS`, `READPLAN`). +When a schema change makes one variant error while another succeeds, the variants produce different output; `-update` runs all of them and each overwrites the other's `output.txt`, so it can silently settle on the passing variant and report `ok` while the golden is actually wrong. Only the non-update run catches this. (Ignore `rejecting_proxy.go: blocking proxy` log lines, which are normal. A test that times out under full parallel load but passes when run alone is a flake, not a regression.) diff --git a/.agents/skills/pr-checklist/SKILL.md b/.agents/skills/pr-checklist/SKILL.md index 88546d02d0e..25a20943d28 100644 --- a/.agents/skills/pr-checklist/SKILL.md +++ b/.agents/skills/pr-checklist/SKILL.md @@ -13,7 +13,7 @@ Before submitting a PR, run these commands to match what CI checks. CI uses the # 2. Linting (CI runs full golangci-lint across all modules, not the diff-only wrapper) ./task lint -# 3. Tests (CI runs with both deployment engines) +# 3. Tests ./task test # 4. If you changed bundle config structs, schema, or direct-engine resource code: diff --git a/AGENTS.md b/AGENTS.md index f22479a7bf3..02b63df8169 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -68,7 +68,7 @@ GIT_EDITOR=true GIT_SEQUENCE_EDITOR=true VISUAL=true GIT_PAGER=cat git rebase or **bundle/** - Core bundle functionality for Declarative Automation Bundles - `bundle/bundle.go` - Main Bundle struct and lifecycle management - `bundle/config/` - Configuration loading, validation, and schema -- `bundle/deploy/` - Deployment logic (Terraform and direct modes) +- `bundle/deploy/` - Deployment logic (file upload, locking, state) - `bundle/mutator/` - Configuration transformation pipeline - `bundle/phases/` - High-level deployment phases @@ -84,9 +84,7 @@ GIT_EDITOR=true GIT_SEQUENCE_EDITOR=true VISUAL=true GIT_PAGER=cat git rebase or **Mutators**: Transform bundle configuration through a pipeline. Located in `bundle/config/mutator/` and `bundle/mutator/`. Each mutator implements the `Mutator` interface. -**Direct vs Terraform Deployment**: The CLI supports two deployment modes controlled by `DATABRICKS_BUNDLE_ENGINE` environment variable: -- `direct` (default) - Direct API calls without Terraform -- `terraform` - Uses Terraform for resource management +**Direct deployment engine**: Bundles are deployed by the direct engine (`bundle/direct/`), which calls the APIs without Terraform. The Terraform engine was removed in v1.20.0: `bundle.engine: terraform` and `DATABRICKS_BUNDLE_ENGINE=terraform` are rejected, and existing Terraform state is migrated to the direct engine before the command runs. # Development Tips diff --git a/acceptance/bundle/help/bundle-deploy/output.txt b/acceptance/bundle/help/bundle-deploy/output.txt index 57fc11b24cb..2b4838b6fa6 100644 --- a/acceptance/bundle/help/bundle-deploy/output.txt +++ b/acceptance/bundle/help/bundle-deploy/output.txt @@ -19,7 +19,7 @@ Flags: --force Force-override Git branch validation. --force-lock Force acquisition of deployment lock. -h, --help help for deploy - --plan string Path to a JSON plan file to apply instead of planning (direct engine only). + --plan string Path to a JSON plan file to apply instead of planning. -q, --quiet count Reduce output: -q prints only the summary, -qq prints only warnings and errors. --select strings Deploy only the specified resource (e.g. 'my_job' or 'jobs.my_job'). Can be repeated or comma-separated. diff --git a/bundle/bundle.go b/bundle/bundle.go index 88ca7193775..5082e8a6c6c 100644 --- a/bundle/bundle.go +++ b/bundle/bundle.go @@ -86,7 +86,7 @@ type Metrics struct { // ResourceState is the direct engine's per-resource deployment state // captured right after the deploy. It carries each resource's state-size in // bytes so deploy telemetry can be derived without re-reading or re-parsing - // the state file. Nil for terraform deploys. + // the state file. ResourceState resourcestate.ExportedResourcesMap } @@ -188,12 +188,10 @@ type Bundle struct { // When non-empty, only the specified resources are included in deployment. Select []string - // MigratingToDirect is set when the direct engine is requested but the existing - // state still uses terraform, so the state is migrated to the direct engine after - // this deploy. Resources that only the direct engine supports are skipped by this - // run rather than rejected: terraform cannot deploy them, and since terraform - // could never have deployed them they are absent from its state. The next deploy, - // which runs on the migrated state, creates them. + // MigratingToDirect is set when the existing state still uses terraform, so it is + // migrated to the direct engine in memory before the command runs (statemgmt.Migrate). + // Deploy and destroy commit the migration (statemgmt.CommitMigration); other commands + // leave the terraform state untouched. MigratingToDirect bool // Quiet is the output verbosity reduction requested via -q/--quiet, which is diff --git a/bundle/config/bundle.go b/bundle/config/bundle.go index a3e8e0952d0..22090bb5e2e 100644 --- a/bundle/config/bundle.go +++ b/bundle/config/bundle.go @@ -46,8 +46,8 @@ type Bundle struct { // Overrides the cluster used for jobs and other supported assets. ClusterId string `json:"cluster_id,omitempty"` - // Engine specifies the deployment engine to use ("terraform" or "direct"). - // Defaults to "direct"; set "terraform" to opt out. + // Engine specifies the deployment engine to use. "direct" is the only supported + // engine and the default; "terraform" is rejected with the removal error. // Can be overridden with the DATABRICKS_BUNDLE_ENGINE environment variable. Engine engine.EngineType `json:"engine,omitempty"` diff --git a/bundle/config/engine/engine.go b/bundle/config/engine/engine.go index 5cc99832940..2df233c4c88 100644 --- a/bundle/config/engine/engine.go +++ b/bundle/config/engine/engine.go @@ -18,7 +18,7 @@ const ( ) // Default is used when the user has not set the value, both for new bundles and -// for existing terraform deployments (which are migrated to it after a deploy). +// for existing terraform deployments (whose state is migrated to it before the command runs). const Default = EngineDirect // TerraformRemovedMessage is the error text shown when a bundle pins the removed diff --git a/bundle/config/experimental.go b/bundle/config/experimental.go index babf44e2b3b..03cd2ba30d4 100644 --- a/bundle/config/experimental.go +++ b/bundle/config/experimental.go @@ -7,7 +7,7 @@ type Experimental struct { // single immutable snapshot rather than being synced individually. When true, // the deployment calls /api/2.0/repos/snapshots with a zip of all files and sets // workspace.file_path and workspace.artifact_path to the returned content-addressed - // path. Only supported with the direct deployment engine. + // path. ImmutableFolder bool `json:"immutable_folder,omitempty"` // By default Python wheel tasks deployed as is to Databricks platform. diff --git a/bundle/config/mutator/drop_empty_strings.go b/bundle/config/mutator/drop_empty_strings.go index 0d4eba75f0b..028351ed943 100644 --- a/bundle/config/mutator/drop_empty_strings.go +++ b/bundle/config/mutator/drop_empty_strings.go @@ -17,10 +17,10 @@ type dropEmptyStrings struct{} // either literally (policy_id: "") or via a variable that resolved to "", so // this must run after variable resolution. // -// Both engines convert the resolved config through convert.ToTyped, which +// The direct engine converts the resolved config through convert.ToTyped, which // force-sends explicitly-set zero values, defeating the omitempty tag. Dropping -// here fixes it uniformly for terraform and direct and makes the result visible -// in `bundle validate -o json`, which serializes the dynamic value. +// here fixes it and makes the result visible in `bundle validate -o json`, which +// serializes the dynamic value. func DropEmptyStrings() bundle.Mutator { return &dropEmptyStrings{} } @@ -43,11 +43,9 @@ func (m *dropEmptyStrings) Apply(ctx context.Context, b *bundle.Bundle) diag.Dia return root, err } - // It seems safe to send an empty description "" to the backend, but for apps - // it causes drift in terraform: the Apps API always returns "" for an unset - // description, so bundle/deploy/terraform/tfdyn/convert_app.go injects it - // anyway. To avoid an unnecessary difference between the direct and terraform - // engines, keep the empty apps description here instead of dropping it. + // Keep an empty apps description instead of dropping it: an app update without + // the field leaves the remote description unchanged, so removing description + // from config would never converge. Force-sending "" clears it. return dyn.MapByPattern(root, dyn.NewPattern(dyn.Key("resources"), dyn.Key("apps"), dyn.AnyKey()), func(_ dyn.Path, app dyn.Value) (dyn.Value, error) { if _, err := dyn.Get(app, "description"); err != nil { return dyn.Set(app, "description", dyn.V("")) diff --git a/bundle/config/mutator/resourcemutator/apply_default_task_source.go b/bundle/config/mutator/resourcemutator/apply_default_task_source.go index c4afd5559e3..30e000b1fb1 100644 --- a/bundle/config/mutator/resourcemutator/apply_default_task_source.go +++ b/bundle/config/mutator/resourcemutator/apply_default_task_source.go @@ -25,9 +25,8 @@ type applyDefaultTaskSource struct{} // Python code added or removed, and so the injected value is not exposed to Python // code. That is the same rationale that first moved this defaulting out of the // pre-Python resource mutators, at the cost of making it invisible in -// `bundle validate`/`summary` and of leaving the direct engine without it -// (terraform got it from tfdyn). Running it here as a normal mutator restores that -// visibility and covers both engines from one place. +// `bundle validate`/`summary` and of leaving the direct engine without it. +// Running it here as a normal mutator restores that visibility. // See https://github.com/databricks/cli/pull/3359 and // https://github.com/databricks/cli/pull/3528. func ApplyDefaultTaskSource() bundle.Mutator { @@ -39,7 +38,6 @@ func (a *applyDefaultTaskSource) Name() string { } // sourceAwareTaskTypes are the task types that support the `source` field. -// Keep in sync with supportedTypeTasks in bundle/deploy/terraform/tfdyn/convert_job.go. // https://docs.databricks.com/api/workspace/jobs/create var sourceAwareTaskTypes = []string{ "dbt_task", diff --git a/bundle/config/resources/model_serving_endpoint.go b/bundle/config/resources/model_serving_endpoint.go index 517ea7fb131..e3bc707cfad 100644 --- a/bundle/config/resources/model_serving_endpoint.go +++ b/bundle/config/resources/model_serving_endpoint.go @@ -15,8 +15,6 @@ type ModelServingEndpoint struct { BaseResource ID string `json:"id,omitempty" bundle:"readonly"` - // This represents the input args for terraform, and will get converted - // to a HCL representation for CRUD serving.CreateServingEndpoint // This is a resource agnostic implementation of permissions for ACLs. diff --git a/bundle/config/resources/registered_model.go b/bundle/config/resources/registered_model.go index eedb33ef3d3..ed06e942f23 100644 --- a/bundle/config/resources/registered_model.go +++ b/bundle/config/resources/registered_model.go @@ -15,8 +15,6 @@ type RegisteredModel struct { BaseResource ID string `json:"id,omitempty" bundle:"readonly"` - // This represents the input args for terraform, and will get converted - // to a HCL representation for CRUD catalog.CreateRegisteredModelRequest // List of grants to apply on this registered model. diff --git a/bundle/direct/bundle_plan.go b/bundle/direct/bundle_plan.go index 61b350e4140..d403ca53f76 100644 --- a/bundle/direct/bundle_plan.go +++ b/bundle/direct/bundle_plan.go @@ -1218,8 +1218,7 @@ func (b *DeploymentBundle) makePlan(ctx context.Context, configRoot *config.Root } // Unescape "$${...}" to a literal "${...}" in the typed state, which is what - // gets deployed and saved. The terraform engine leaves the escape in place for - // Terraform itself to unescape; the direct engine has to do it here. + // gets deployed and saved. // // This runs on the typed state rather than the dynamic config so that // extractReferences below still sees the escaped form and does not mistake diff --git a/bundle/direct/dresources/adapter.go b/bundle/direct/dresources/adapter.go index 74c19c08445..80ece24b2be 100644 --- a/bundle/direct/dresources/adapter.go +++ b/bundle/direct/dresources/adapter.go @@ -79,7 +79,7 @@ type IResource interface { // [Optional] IsEmptyState reports that newState describes no resource at all: the planner // omits the node instead of planning a create, and apply drops the state entry instead of - // persisting one, so both engines converge on "this node does not exist". + // persisting one, so plan and apply converge on "this node does not exist". // Example: func (*ResourceGrants) IsEmptyState(state *GrantsState) bool IsEmptyState(newState any) bool diff --git a/bundle/internal/schema/annotations.yml b/bundle/internal/schema/annotations.yml index f31234d00d2..710e4b8373f 100644 --- a/bundle/internal/schema/annotations.yml +++ b/bundle/internal/schema/annotations.yml @@ -622,7 +622,7 @@ resources: "$fields": "libraries": "description": |- - A list of libraries to install on the cluster. Installed via the Libraries API after the cluster is created. Only supported in direct deployment mode. + A list of libraries to install on the cluster. Installed via the Libraries API after the cluster is created. "lifecycle": "description": |- Settings that control the deployment lifecycle of the resource, such as preventing it from being destroyed. @@ -632,7 +632,7 @@ resources: Lifecycle setting to prevent the resource from being destroyed. "started": "description": |- - Lifecycle setting to deploy the resource in started mode. Only supported for apps, clusters, and sql_warehouses in direct deployment mode. + Lifecycle setting to deploy the resource in started mode. Only supported for apps, clusters, and sql_warehouses. "permissions": "description": |- The permissions to apply to this resource. diff --git a/bundle/phases/telemetry.go b/bundle/phases/telemetry.go index 4b6d705eba9..1b6e22f5298 100644 --- a/bundle/phases/telemetry.go +++ b/bundle/phases/telemetry.go @@ -300,11 +300,11 @@ func LogDeployTelemetry(ctx context.Context, b *bundle.Bundle, errMsg string) { b.Metrics.SetBoolValue(metrics.AiRuntimeTaskMultitask, airMultitask) } - // Record whether the deprecated terraform engine was explicitly opted into, - // separately per source. Only emitted when true; absence means "not opted in - // via this source". An invalid env value has already failed the command - // upstream via ResolveEngineSetting, so treating the FromEnv error as - // "not terraform" here cannot mask a real terraform opt-in. + // Record whether the removed terraform engine was still requested (the command + // then fails with the removal error), separately per source. Only emitted when + // true; absence means "not requested via this source". An invalid env value has + // already failed the command upstream via ResolveEngineSetting, so treating the + // FromEnv error as "not terraform" here cannot mask a real terraform request. if b.Config.Bundle.Engine == engine.EngineTerraform { b.Metrics.SetBoolValue(metrics.EngineTerraformConfig, true) } diff --git a/bundle/schema/jsonschema.json b/bundle/schema/jsonschema.json index 21aa6f57cc0..f52e8059ee4 100644 --- a/bundle/schema/jsonschema.json +++ b/bundle/schema/jsonschema.json @@ -535,7 +535,7 @@ "x-databricks-launch-stage": "GA" }, "libraries": { - "description": "A list of libraries to install on the cluster. Installed via the Libraries API after the cluster is created. Only supported in direct deployment mode.", + "description": "A list of libraries to install on the cluster. Installed via the Libraries API after the cluster is created.", "$ref": "#/$defs/slice/github.com/databricks/databricks-sdk-go/service/compute.Library" }, "lifecycle": { @@ -1580,7 +1580,7 @@ "$ref": "#/$defs/bool" }, "started": { - "description": "Lifecycle setting to deploy the resource in started mode. Only supported for apps, clusters, and sql_warehouses in direct deployment mode.", + "description": "Lifecycle setting to deploy the resource in started mode. Only supported for apps, clusters, and sql_warehouses.", "$ref": "#/$defs/bool" } }, diff --git a/bundle/statemgmt/direct_migration.go b/bundle/statemgmt/direct_migration.go index a84b5c1f804..00aa30fd751 100644 --- a/bundle/statemgmt/direct_migration.go +++ b/bundle/statemgmt/direct_migration.go @@ -276,9 +276,8 @@ func convertTFStateToDirect(ctx context.Context, b *bundle.Bundle, tfState *migr return tempStatePath, false, nil, errors.New("failed to apply secret scope fixups") } - // b.Config has been modified by terraform.Interpolate which converts bundle-style - // references (${resources.pipelines.x.id}) to terraform-style (${databricks_pipeline.x.id}). - // BuildStateFromTF expects ${resources.*} references, so reverse the interpolation first. + // The config may use terraform-style references (${databricks_pipeline.x.id}). + // BuildStateFromTF expects ${resources.*} references, so rewrite them first. uninterpolatedRoot, err := reverseInterpolate(b.Config.Value()) if err != nil { return tempStatePath, false, nil, fmt.Errorf("failed to reverse interpolation: %w", err) diff --git a/bundle/statemgmt/reverse_interpolate.go b/bundle/statemgmt/reverse_interpolate.go index 3e74caee341..d133c0e0f7d 100644 --- a/bundle/statemgmt/reverse_interpolate.go +++ b/bundle/statemgmt/reverse_interpolate.go @@ -8,8 +8,7 @@ import ( "github.com/databricks/cli/libs/dyn/dynvar" ) -// reverseInterpolate reverses the terraform.Interpolate transformation. -// It converts terraform-style resource references back to bundle-style. +// reverseInterpolate converts terraform-style resource references to bundle-style. // Example: ${databricks_pipeline.my_etl.id} → ${resources.pipelines.my_etl.id} func reverseInterpolate(root dyn.Value) (dyn.Value, error) { return dynvar.Resolve(root, func(path dyn.Path) (dyn.Value, error) { diff --git a/cmd/apps/deploy_bundle.go b/cmd/apps/deploy_bundle.go index 6494efe45f0..b390f45e132 100644 --- a/cmd/apps/deploy_bundle.go +++ b/cmd/apps/deploy_bundle.go @@ -78,7 +78,7 @@ func BundleDeployOverrideWithWrapper(wrapError ErrorWrapper) func(*cobra.Command deployCmd.Flags().BoolVar(&opts.autoApprove, "auto-approve", false, "Skip interactive approvals that might be required for deployment.") deployCmd.Flags().MarkDeprecated("compute-id", "use --cluster-id instead") deployCmd.Flags().BoolVar(&opts.verbose, "verbose", false, "Enable verbose output.") - deployCmd.Flags().StringVar(&opts.readPlanPath, "plan", "", "Path to a JSON plan file to apply instead of planning (direct engine only).") + deployCmd.Flags().StringVar(&opts.readPlanPath, "plan", "", "Path to a JSON plan file to apply instead of planning.") // Verbose flag currently only affects file sync output, it's used by the vscode extension deployCmd.Flags().MarkHidden("verbose") deployCmd.Flags().BoolVar(&opts.skipValidation, "skip-validation", false, "Skip project validation (build, typecheck, lint)") diff --git a/cmd/apps/import.go b/cmd/apps/import.go index ea3fce67e93..e79e9ef4353 100644 --- a/cmd/apps/import.go +++ b/cmd/apps/import.go @@ -52,8 +52,8 @@ func newImportCommand() *cobra.Command { Long: `(Experimental) Import an existing Databricks app and convert it to a bundle configuration. This command creates a new bundle directory with the app configuration, downloads -the app source code, binds the bundle to the existing app, and deploys it using -direct deployment mode. This allows you to manage the app as code going forward. +the app source code, binds the bundle to the existing app, and deploys it. This +allows you to manage the app as code going forward. The command will: 1. Create an empty bundle folder with databricks.yml diff --git a/cmd/bundle/deploy.go b/cmd/bundle/deploy.go index f248111e8b0..6f1dce84d32 100644 --- a/cmd/bundle/deploy.go +++ b/cmd/bundle/deploy.go @@ -42,7 +42,7 @@ See https://docs.databricks.com/en/dev-tools/bundles/index.html for more informa cmd.Flags().MarkDeprecated("compute-id", "use --cluster-id instead") cmd.Flags().BoolVar(&verbose, "verbose", false, "Enable verbose output.") cmd.Flags().CountVarP(&quiet, "quiet", "q", "Reduce output: -q prints only the summary, -qq prints only warnings and errors.") - cmd.Flags().StringVar(&readPlanPath, "plan", "", "Path to a JSON plan file to apply instead of planning (direct engine only).") + cmd.Flags().StringVar(&readPlanPath, "plan", "", "Path to a JSON plan file to apply instead of planning.") cmd.Flags().StringSliceVar(&selectResources, "select", nil, "Deploy only the specified resource (e.g. 'my_job' or 'jobs.my_job'). Can be repeated or comma-separated.") // Verbose flag currently only affects file sync output, it's used by the vscode extension cmd.Flags().MarkHidden("verbose") diff --git a/docs/direct.md b/docs/direct.md index 49ce42feff7..8bec8824a54 100644 --- a/docs/direct.md +++ b/docs/direct.md @@ -1,6 +1,6 @@ ## Status -Status: Generally Available (GA). The direct engine is the default; existing Terraform deployments are migrated to it automatically. +Status: Generally Available (GA). The direct engine is the only deployment engine since Databricks CLI v1.20.0; existing Terraform deployments are migrated to it automatically. Known issues: https://github.com/databricks/cli/issues?q=state%3Aopen%20label%3Aengine%2Fdirect ## Reporting bugs @@ -12,9 +12,9 @@ Please ensure you are on the latest version of CLI when reporting an issue. ## Background Declarative Automation Bundles were originally implemented on top of databricks-terraform-provider. -Since version 0.279.0 DABs support two different deployment engines: "terraform" and "direct". -The latter does not make use of Terraform. -It is intended to be a drop-in replacement and will become the only engine we support in 2026\. +Version 0.279.0 added a second deployment engine, "direct", which does not make use of Terraform. +It was designed as a drop-in replacement, and since v1.20.0 it is the only engine: the Terraform engine was removed +(see https://github.com/databricks/cli/issues/6765). ## Advantages @@ -37,34 +37,30 @@ There are known issues, see https://github.com/databricks/cli/issues?q=state%3Ao ### Migrating the existing deployment The direct engine uses its own state file, also JSON, but with a different schema from terraform state file. -Since v1.14.0, after deploying with terraform, automatic migration to direct engine is attempted. If successful, it replaces terraform state with direct state so following operations use direct engine. +When a bundle still has Terraform state (`.databricks/bundle//terraform/terraform.tfstate` or its copy in the workspace), the CLI converts it to direct state in memory before running the command. Commands that only read state (e.g. `bundle plan`, `bundle summary`) work on the converted state without writing it. `bundle deploy` and `bundle destroy` commit the migration: the direct state is written locally and to the workspace, and the Terraform state is kept as `terraform.tfstate.backup`. + +If the automatic migration fails, the command fails with an error; please report it to dabs-feedback@databricks.com. To keep deploying with Terraform in the meantime, use Databricks CLI v1.19.x, the last version that ships the Terraform engine. #### Manual migration via "bundle deployment migrate" command -Migrating via this command is still possible, but no longer necessary. The command writes local state only, so it doubles as a way to preview the direct engine without committing to it: run `bundle plan` to check that it works, then `bundle deploy` to commit the migration — or, before that deploy, remove `.databricks` to undo it. +Migrating via this command is still possible, but no longer necessary. The command writes local state only and does not push it to the workspace. The full sequence of operations: -1. Perform full deployment with Terraform: `databricks bundle deploy -t my_target` -2. Migrate state file locally: `databricks bundle deployment migrate -t my_target` -3. Verify that migration was successful: bundle plan should work and should not show any changes to be planned: `databricks bundle plan -t my_target` -4. If not satisfied with the result, remove new state file and restore terraform state file from the backup: +1. Migrate state file locally: `databricks bundle deployment migrate -t my_target` +2. Verify that migration was successful: bundle plan should work and should not show any changes to be planned: `databricks bundle plan -t my_target` +3. If not satisfied with the result, remove new state file and restore terraform state file from the backup: ``` -mv .databricks/bundle/my_target/terraform/tfstate.json.backup .databricks/bundle/my_target/terraform/tfstate.json +mv .databricks/bundle/my_target/terraform/terraform.tfstate.backup .databricks/bundle/my_target/terraform/terraform.tfstate rm .databricks/bundle/my_target/resources.json ``` -5. If satisfied with the result, do a deployment to synchronize the state file to the workspace: `databricks bundle deploy -t my_target` +4. If satisfied with the result, do a deployment to synchronize the state file to the workspace: `databricks bundle deploy -t my_target` ### Using on new bundles -New bundles use the direct engine by default, so no configuration is needed. The migrate command does not apply here; it is only for bundles previously deployed with Terraform. - -To opt out and keep using Terraform, you have two options: - -- Set bundle.engine OR targets.\*.engine to "terraform" in your databricks.yml -- Set DATABRICKS_BUNDLE_ENGINE=terraform env var before the deployment. +New bundles use the direct engine, so no configuration is needed. The migrate command does not apply here; it is only for bundles previously deployed with Terraform. -If both are provided, the config takes precedence over the env var. +Setting `bundle.engine` (or `targets.*.engine`) or `DATABRICKS_BUNDLE_ENGINE` to "terraform" is an error. To keep using Terraform, use Databricks CLI v1.19.x. ## Differences from terraform @@ -84,7 +80,7 @@ The consequences of this are: ### $resources references lookup -Note, the most common use of $resources is resolving ID ($resources.jobs.foo.id) which should behave identically between terraform and direct engine. The differences can happen in other fields. +Note, the most common use of $resources is resolving ID ($resources.jobs.foo.id) which behaves identically between terraform and direct engine. The differences can happen in other fields. The resolution of $resources references in direct engine (e.g. $resources.pipelines.foo.name) is performed in two steps: diff --git a/python/databricks/bundles/apps/_models/lifecycle_with_started.py b/python/databricks/bundles/apps/_models/lifecycle_with_started.py index 3ee1d2a01b3..f9cb9b8d49c 100644 --- a/python/databricks/bundles/apps/_models/lifecycle_with_started.py +++ b/python/databricks/bundles/apps/_models/lifecycle_with_started.py @@ -22,7 +22,7 @@ class LifecycleWithStarted: started: VariableOrOptional[bool] = None """ - Lifecycle setting to deploy the resource in started mode. Only supported for apps, clusters, and sql_warehouses in direct deployment mode. + Lifecycle setting to deploy the resource in started mode. Only supported for apps, clusters, and sql_warehouses. """ @classmethod @@ -43,7 +43,7 @@ class LifecycleWithStartedDict(TypedDict, total=False): started: VariableOrOptional[bool] """ - Lifecycle setting to deploy the resource in started mode. Only supported for apps, clusters, and sql_warehouses in direct deployment mode. + Lifecycle setting to deploy the resource in started mode. Only supported for apps, clusters, and sql_warehouses. """ diff --git a/python/databricks/bundles/clusters/_models/cluster.py b/python/databricks/bundles/clusters/_models/cluster.py index 042a2c14a52..98c9d9a6acb 100644 --- a/python/databricks/bundles/clusters/_models/cluster.py +++ b/python/databricks/bundles/clusters/_models/cluster.py @@ -245,7 +245,7 @@ class Cluster(Resource): libraries: VariableOrList[Library] = field(default_factory=list) """ - A list of libraries to install on the cluster. Installed via the Libraries API after the cluster is created. Only supported in direct deployment mode. + A list of libraries to install on the cluster. Installed via the Libraries API after the cluster is created. """ lifecycle: VariableOrOptional[LifecycleWithStarted] = None @@ -538,7 +538,7 @@ class ClusterDict(TypedDict, total=False): libraries: VariableOrList[LibraryParam] """ - A list of libraries to install on the cluster. Installed via the Libraries API after the cluster is created. Only supported in direct deployment mode. + A list of libraries to install on the cluster. Installed via the Libraries API after the cluster is created. """ lifecycle: VariableOrOptional[LifecycleWithStartedParam] diff --git a/python/databricks/bundles/clusters/_models/lifecycle_with_started.py b/python/databricks/bundles/clusters/_models/lifecycle_with_started.py index 3ee1d2a01b3..f9cb9b8d49c 100644 --- a/python/databricks/bundles/clusters/_models/lifecycle_with_started.py +++ b/python/databricks/bundles/clusters/_models/lifecycle_with_started.py @@ -22,7 +22,7 @@ class LifecycleWithStarted: started: VariableOrOptional[bool] = None """ - Lifecycle setting to deploy the resource in started mode. Only supported for apps, clusters, and sql_warehouses in direct deployment mode. + Lifecycle setting to deploy the resource in started mode. Only supported for apps, clusters, and sql_warehouses. """ @classmethod @@ -43,7 +43,7 @@ class LifecycleWithStartedDict(TypedDict, total=False): started: VariableOrOptional[bool] """ - Lifecycle setting to deploy the resource in started mode. Only supported for apps, clusters, and sql_warehouses in direct deployment mode. + Lifecycle setting to deploy the resource in started mode. Only supported for apps, clusters, and sql_warehouses. """ diff --git a/python/databricks/bundles/sql_warehouses/_models/lifecycle_with_started.py b/python/databricks/bundles/sql_warehouses/_models/lifecycle_with_started.py index 3ee1d2a01b3..f9cb9b8d49c 100644 --- a/python/databricks/bundles/sql_warehouses/_models/lifecycle_with_started.py +++ b/python/databricks/bundles/sql_warehouses/_models/lifecycle_with_started.py @@ -22,7 +22,7 @@ class LifecycleWithStarted: started: VariableOrOptional[bool] = None """ - Lifecycle setting to deploy the resource in started mode. Only supported for apps, clusters, and sql_warehouses in direct deployment mode. + Lifecycle setting to deploy the resource in started mode. Only supported for apps, clusters, and sql_warehouses. """ @classmethod @@ -43,7 +43,7 @@ class LifecycleWithStartedDict(TypedDict, total=False): started: VariableOrOptional[bool] """ - Lifecycle setting to deploy the resource in started mode. Only supported for apps, clusters, and sql_warehouses in direct deployment mode. + Lifecycle setting to deploy the resource in started mode. Only supported for apps, clusters, and sql_warehouses. """