Skip to content

bundle: always error on references to Terraform-only fields #6937

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
denik merged 2 commits into from
Oct 6, 2026
Merged

bundle: always error on references to Terraform-only fields #6937

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
8ab9370
bundle: always error on references to Terraform-only fields
denik Oct 5, 2026
08c1ddf
Update the comment on the post-initialize engine resolution
denik Oct 6, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 9 additions & 0 deletions acceptance/bundle/resource_deps/tf_path_only_error/out.validate-terraform-env.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
Error: the Terraform deployment engine has been removed in Databricks CLI v1.20.0

Remove the "bundle.engine" setting (or set it to "direct") to deploy with the direct engine; existing Terraform state is migrated automatically. To keep using Terraform, revert to Databricks CLI v1.19.x.
See https://docs.databricks.com/dev-tools/bundles/direct for details

@janniklasrose janniklasrose Oct 6, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is awkward since the bundle doesn't contain bundle.engine but TF was forced via the environment variable. Out of scope of this PR but we should fix that anyway

@denik denik Oct 6, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good point, noted as follow up.

@denik denik Oct 6, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

#6951


Name: test-bundle
Target: default

Found 1 error
3 changes: 3 additions & 0 deletions acceptance/bundle/resource_deps/tf_path_only_error/script
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,3 +3,6 @@ errcode $CLI bundle plan > out.plan.txt 2>&1
errcode $CLI bundle deploy > out.deploy.txt 2>&1
print_telemetry_bool_values > out.telemetry.txt
rm "$OUT_REQUESTS"

# The engine removal error comes before the Terraform-only reference error.
DATABRICKS_BUNDLE_ENGINE=terraform musterr $CLI bundle validate > out.validate-terraform-env.txt 2>&1
33 changes: 6 additions & 27 deletions bundle/config/validate/tf_only_references.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,6 @@ import (
"strings"

"github.com/databricks/cli/bundle"
"github.com/databricks/cli/bundle/config/engine"
"github.com/databricks/cli/bundle/terraform_dabs_map"
"github.com/databricks/cli/libs/diag"
"github.com/databricks/cli/libs/dyn"
Expand All @@ -18,10 +17,7 @@ type tfOnlyReferences struct{}

// TFOnlyReferences validates that no cross-resource references point to
// Terraform-only fields (fields that exist in TF schema but have no DABs equivalent).
//
// In direct mode this is an error because the direct engine cannot resolve such
// references at deploy time. In Terraform mode it is a warning because the
// reference will break if the bundle is later migrated to the direct engine.
// The direct engine cannot resolve such references at deploy time.
func TFOnlyReferences() bundle.Mutator {
return &tfOnlyReferences{}
}
Expand All @@ -30,16 +26,7 @@ func (m *tfOnlyReferences) Name() string {
return "validate:tf_only_references"
}

func (m *tfOnlyReferences) Apply(ctx context.Context, b *bundle.Bundle) diag.Diagnostics {
// Resolve effective engine: config takes precedence over env var.
effectiveEngine := b.Config.Bundle.Engine
if effectiveEngine == engine.EngineNotSet {
if envEngine, err := engine.FromEnv(ctx); err == nil {
effectiveEngine = envEngine
}
}
isDirect := effectiveEngine.IsDirect()

func (m *tfOnlyReferences) Apply(_ context.Context, b *bundle.Bundle) diag.Diagnostics {
var diags diag.Diagnostics

// Walk the entire config looking for ${resources.*} references.
Expand All @@ -52,7 +39,7 @@ func (m *tfOnlyReferences) Apply(ctx context.Context, b *bundle.Bundle) diag.Dia
if !strings.HasPrefix(r, "resources.") {
continue
}
if d := checkTFOnlyReference(r, v.Location(), isDirect); d != nil {
if d := checkTFOnlyReference(r, v.Location()); d != nil {
diags = append(diags, *d)
}
}
Expand All @@ -69,7 +56,7 @@ func (m *tfOnlyReferences) Apply(ctx context.Context, b *bundle.Bundle) diag.Dia
// checkTFOnlyReference checks a single reference string like
// "resources.jobs.src.always_running" and returns a diagnostic when it refers
// to a TF-only field, or nil otherwise.
func checkTFOnlyReference(ref string, loc dyn.Location, isDirect bool) *diag.Diagnostic {
func checkTFOnlyReference(ref string, loc dyn.Location) *diag.Diagnostic {
p, err := dyn.NewPathFromString(ref)
// Need at least resources.<group>.<name>.<field>
if err != nil || len(p) < 4 || p[0].Key() != "resources" {
Expand All @@ -92,17 +79,9 @@ func checkTFOnlyReference(ref string, loc dyn.Location, isDirect bool) *diag.Dia
return nil
}

if isDirect {
return &diag.Diagnostic{
Severity: diag.Error,
Summary: fmt.Sprintf("%q: Terraform-only field; cross-resource references to Terraform-only fields are not supported by the direct engine", ref),
Locations: []dyn.Location{loc},
}
}

return &diag.Diagnostic{
Severity: diag.Warning,
Summary: fmt.Sprintf("%q: Terraform-only field; this reference will not work with the direct engine", ref),
Severity: diag.Error,
Summary: fmt.Sprintf("%q: Terraform-only field; cross-resource references to Terraform-only fields are not supported by the direct engine", ref),
Locations: []dyn.Location{loc},
}
}
27 changes: 4 additions & 23 deletions bundle/config/validate/tf_only_references_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,12 +5,10 @@ import (

"github.com/databricks/cli/bundle"
"github.com/databricks/cli/bundle/config"
"github.com/databricks/cli/bundle/config/engine"
"github.com/databricks/cli/bundle/config/resources"
"github.com/databricks/cli/bundle/internal/bundletest"
"github.com/databricks/cli/libs/diag"
"github.com/databricks/cli/libs/dyn"
"github.com/databricks/cli/libs/env"
"github.com/databricks/databricks-sdk-go/service/jobs"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
Expand All @@ -31,43 +29,27 @@ func makeBundle(t *testing.T) *bundle.Bundle {
return b
}

func TestTFOnlyReferences_DirectError(t *testing.T) {
func TestTFOnlyReferences_Error(t *testing.T) {
b := makeBundle(t)
bundletest.Mutate(t, b, func(v dyn.Value) (dyn.Value, error) {
return dyn.Set(v, "resources.jobs.dst.name", dyn.V("${resources.jobs.src.always_running}"))
})

ctx := env.Set(t.Context(), engine.EnvVar, "direct")
diags := TFOnlyReferences().Apply(ctx, b)
diags := TFOnlyReferences().Apply(t.Context(), b)
require.Len(t, diags, 1)
assert.Equal(t, diag.Error, diags[0].Severity)
assert.Contains(t, diags[0].Summary, "resources.jobs.src.always_running")
assert.Contains(t, diags[0].Summary, "Terraform-only field")
}

func TestTFOnlyReferences_TerraformWarning(t *testing.T) {
b := makeBundle(t)
bundletest.Mutate(t, b, func(v dyn.Value) (dyn.Value, error) {
return dyn.Set(v, "resources.jobs.dst.name", dyn.V("${resources.jobs.src.always_running}"))
})

ctx := env.Set(t.Context(), engine.EnvVar, "terraform")
diags := TFOnlyReferences().Apply(ctx, b)
require.Len(t, diags, 1)
assert.Equal(t, diag.Warning, diags[0].Severity)
assert.Contains(t, diags[0].Summary, "resources.jobs.src.always_running")
assert.Contains(t, diags[0].Summary, "Terraform-only field")
}

func TestTFOnlyReferences_NormalReference(t *testing.T) {
b := makeBundle(t)
bundletest.Mutate(t, b, func(v dyn.Value) (dyn.Value, error) {
// "name" is not a TF-only field; no diagnostic expected.
return dyn.Set(v, "resources.jobs.dst.name", dyn.V("${resources.jobs.src.name}"))
})

ctx := env.Set(t.Context(), engine.EnvVar, "direct")
diags := TFOnlyReferences().Apply(ctx, b)
diags := TFOnlyReferences().Apply(t.Context(), b)
assert.Empty(t, diags)
}

Expand All @@ -78,7 +60,6 @@ func TestTFOnlyReferences_RenamedField(t *testing.T) {
return dyn.Set(v, "resources.jobs.dst.name", dyn.V("${resources.jobs.src.git_source[0].branch}"))
})

ctx := env.Set(t.Context(), engine.EnvVar, "direct")
diags := TFOnlyReferences().Apply(ctx, b)
diags := TFOnlyReferences().Apply(t.Context(), b)
assert.Empty(t, diags)
}
4 changes: 2 additions & 2 deletions bundle/phases/initialize.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -201,8 +201,8 @@ func Initialize(ctx context.Context, b *bundle.Bundle) {
mutator.ValidateJobRunTriggers(),

// Reads (dynamic): * (strings) (searches for ${resources.*} references)
// Warns (TF engine) or errors (direct engine) when a cross-resource reference
// points to a Terraform-only field with no DABs equivalent.
// Errors when a cross-resource reference points to a Terraform-only field
// with no DABs equivalent.
validate.TFOnlyReferences(),

// Reads (typed): b.Config.Permissions (checks if current user or their groups have CAN_MANAGE permissions)
Expand Down
19 changes: 15 additions & 4 deletions cmd/bundle/utils/process.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -167,6 +167,16 @@ func ProcessBundleRet(cmd *cobra.Command, opts ProcessOptions) (b *bundle.Bundle
cmd.SetContext(ctx)
}

// Reject an invalid or removed DATABRICKS_BUNDLE_ENGINE before initialization, so
// the removal error comes before any initialize-time validation (such as
// validate.TFOnlyReferences). A terraform bundle.engine is reported during
// initialization by validate.ValidateEngine, with its location.
if b.Config.Bundle.Engine == engine.EngineNotSet {
if _, err := ResolveEngineSetting(ctx, b); err != nil {
return b, nil, err
}
}

if !opts.SkipInitialize {
t0 := time.Now()
phases.Initialize(ctx, b)
Expand Down Expand Up @@ -198,10 +208,11 @@ func ProcessBundleRet(cmd *cobra.Command, opts ProcessOptions) (b *bundle.Bundle
}
}

// Resolve engine setting up front so a garbage DATABRICKS_BUNDLE_ENGINE
// value fails every bundle command instead of only the ones that read
// state. The resolver is cheap (config lookup + env var read); no reason
// to gate it on state-touching options.
// Resolve the engine used for state and telemetry. An invalid or removed
// DATABRICKS_BUNDLE_ENGINE was already rejected before initialization and a
// bad bundle.engine by validate.ValidateEngine, so this only fails for
// commands that skip initialization (bundle run -- <cmd>) with a bad
// bundle.engine.
requiredEngine, err := ResolveEngineSetting(ctx, b)
if err != nil {
return b, nil, err
Expand Down
Loading