Skip to content

Allow bundle run with engine: terraform #6972

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
denik merged 4 commits into from
Oct 7, 2026
Merged

Allow bundle run with engine: terraform #6972

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
75ead0d
acc: bundle run with terraform state uses exact job id; rejects engin…
denik Oct 7, 2026
74df178
Allow bundle run with engine: terraform
denik Oct 7, 2026
0a6affe
acc: bundle run works with engine: terraform
denik Oct 7, 2026
1f43509
Inline allowTerraformEngineConfig in bundle run
denik Oct 7, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 1 addition & 2 deletions acceptance/bundle/migrate/auto/run-no-commit/databricks.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,4 @@
bundle:
name: test-bundle
bundle: {name: test-bundle}

resources:
jobs:
Expand Down
33 changes: 32 additions & 1 deletion acceptance/bundle/migrate/auto/run-no-commit/output.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@
=== bundle run on direct migrates the terraform state IN MEMORY (no commit) to resolve the job
>>> DATABRICKS_BUNDLE_ENGINE=direct errcode [CLI] bundle run test_job --no-wait
Notice: migrating your bundle to direct deployment engine (https://github.com/databricks/cli/issues/6765).
Run URL: [DATABRICKS_URL]/jobs/[NUMID]/runs/[NUMID]?o=[NUMID]
Run URL: [DATABRICKS_URL]/jobs/[TEST_JOB_ID]/runs/[NUMID]?o=[NUMID]


>>> print_migration_telemetry
Expand All @@ -18,3 +18,34 @@ Run URL: [DATABRICKS_URL]/jobs/[NUMID]/runs/[NUMID]?o=[NUMID]
>>> MSYS_NO_PATHCONV=1 [CLI] workspace list /Workspace/Users/[USERNAME]/.bundle/test-bundle/default/state
ID Type Language Path
[NUMID] FILE /Workspace/Users/[USERNAME]/.bundle/test-bundle/default/state/terraform.tfstate

=== bundle run works with engine: terraform: warns and runs the job from the terraform state
>>> update_file.py databricks.yml {name: test-bundle} {name: test-bundle, engine: terraform}

>>> [CLI] bundle run test_job --no-wait
Warning: the Terraform deployment engine has been removed in Databricks CLI v1.20.0
in databricks.yml:1:37

Remove the "bundle.engine" setting (or set it to "direct") to deploy with the direct engine; existing Terraform state is migrated automatically. To keep using Terraform, revert to Databricks CLI v1.19.x.
See https://docs.databricks.com/dev-tools/bundles/direct for details

Notice: migrating your bundle to direct deployment engine (https://github.com/databricks/cli/issues/6765).
Run URL: [DATABRICKS_URL]/jobs/[TEST_JOB_ID]/runs/[NUMID]?o=[NUMID]


>>> print_requests.py //jobs
{
"headers": {
"User-Agent": [
"cli/[CLI_VERSION] databricks-sdk-go/[SDK_VERSION] go/[GO_VERSION] os/[OS] cmd/bundle_run cmd-exec-id/[UUID] interactive/none engine/direct auth/pat"
]
},
"method": "POST",
"path": "/api/2.2/jobs/run-now",
"body": {
"job_id": [TEST_JOB_ID]
}
}

>>> find.py \.databricks/.*(\.tfstate|resources.*\.json|\.wal)
.databricks/bundle/default/terraform/terraform.tfstate
8 changes: 8 additions & 0 deletions acceptance/bundle/migrate/auto/run-no-commit/script
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,7 @@ export DATABRICKS_BUNDLE_ENGINE=

title "Replay a prior terraform deployment"
trace replay_tfstate.py terraform-requests.json terraform.tfstate
replace_ids.py
rm -f "$OUT_REQUESTS"

title "bundle run on direct migrates the terraform state IN MEMORY (no commit) to resolve the job"
Expand All @@ -17,3 +18,10 @@ title "Nothing durable written: still on terraform (local state files + remote),
# cache under .terraform/ is excluded by matching only state files (tfstate/.backup, resources*.json, WAL).
trace find.py '\.databricks/.*(\.tfstate|resources.*\.json|\.wal)'
trace MSYS_NO_PATHCONV=1 $CLI workspace list "/Workspace/Users/${CURRENT_USER_NAME}/.bundle/test-bundle/default/state"

title "bundle run works with engine: terraform: warns and runs the job from the terraform state"
# Set after the replay: replay_tfstate.py runs bundle validate, which rejects engine: terraform.
trace update_file.py databricks.yml "{name: test-bundle}" "{name: test-bundle, engine: terraform}"
trace $CLI bundle run test_job --no-wait
trace print_requests.py //jobs
trace find.py '\.databricks/.*(\.tfstate|resources.*\.json|\.wal)'
7 changes: 7 additions & 0 deletions cmd/bundle/run.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -138,6 +138,9 @@ Example usage:
if cmd.ArgsLenAtDash() == 0 && len(args) > 0 {
b, err := utils.ProcessBundle(cmd, utils.ProcessOptions{
SkipInitialize: true,
InitFunc: func(b *bundle.Bundle) {
b.AllowTerraformEngineConfig = true
},
})
if err != nil {
return err
Expand All @@ -154,6 +157,10 @@ Example usage:
_, _, err := utils.ProcessBundleRet(cmd, utils.ProcessOptions{
AlwaysPull: true,
ErrorOnEmptyState: true,
// Let users run a bundle that pins the removed Terraform engine.
InitFunc: func(b *bundle.Bundle) {
b.AllowTerraformEngineConfig = true
},
PostInitFunc: func(ctx context.Context, b *bundle.Bundle) error {
var err error
key, runArgs, err = resolveRunArgument(ctx, b, args)
Expand Down
Loading