Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion acceptance/bundle/migrate/command/basic/output.txt
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
>>> musterr [CLI] bundle deployment migrate
Error: This command migrates the existing Terraform state file (terraform.tfstate) to a direct deployment state file (resources.json). However, no existing local or remote state was found.

To start using direct engine, set "engine: direct" under bundle in your databricks.yml or deploy with DATABRICKS_BUNDLE_ENGINE=direct env var set.
New bundles use the direct engine automatically, no migration is needed.

>>> replay_tfstate.py terraform-requests.json terraform.tfstate -t dev

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -2,4 +2,4 @@
>>> musterr [CLI] bundle deployment migrate
Error: This command migrates the existing Terraform state file (terraform.tfstate) to a direct deployment state file (resources.json). However, no existing local or remote state was found.

To start using direct engine, set "engine: direct" under bundle in your databricks.yml or deploy with DATABRICKS_BUNDLE_ENGINE=direct env var set.
New bundles use the direct engine automatically, no migration is needed.
2 changes: 1 addition & 1 deletion acceptance/internal/selection/cmd/main.go
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@
//
// Each line is a `go test -run` pattern, so a selected test can be run as printed:
//
// go test ./acceptance -run 'TestAccept/^bundle$/^invariant$/^no_drift$/^DATABRICKS_BUNDLE_ENGINE=direct$/^INPUT_CONFIG=job.yml.tmpl$'
// go test ./acceptance -run 'TestAccept/^bundle$/^invariant$/^no_drift$/^INPUT_CONFIG=job.yml.tmpl$'
//
// Every element is anchored, because a -run element is a regexp matched against one level of
// the test name: unanchored, `no_drift` would also run no_drift_extra, and `DMS=` would also
Expand Down
4 changes: 2 additions & 2 deletions bundle/config/bundle.go
Original file line number Diff line number Diff line change
Expand Up @@ -23,8 +23,8 @@ type Bundle struct {
// DEPRECATED. Left for backward compatibility with Target
Environment string `json:"environment,omitempty" bundle:"readonly"`

// Terraform holds configuration related to Terraform.
// For example, where to find the binary, which version to use, etc.
// DEPRECATED. The Terraform engine was removed, so this setting is ignored
// (a warning is reported if it is set). Kept so existing configs still parse.
Terraform *Terraform `json:"terraform,omitempty" bundle:"readonly"`

// Force-override Git branch validation.
Expand Down
5 changes: 2 additions & 3 deletions bundle/direct/dresources/catalog.go
Original file line number Diff line number Diff line change
Expand Up @@ -53,9 +53,8 @@ func (r *ResourceCatalog) DoCreate(ctx context.Context, config *catalog.CreateCa
// See schemaForceSend. Verified against a real workspace: {"comment": ""} clears it.
//
// CustomMaxRetentionHours is force-sent here even though schemaForceSend deliberately omits it:
// that omission is for terraform parity, but catalogs are direct-only (no terraform converter, see
// validate_direct_only_resources), so there is no cross-engine payload to diverge from. Verified
// against a real workspace: {"custom_max_retention_hours": 0} clears it (an empty PATCH is otherwise
// that omission was for parity with the removed Terraform engine, which does not apply to catalogs
// anyway. Verified against a real workspace: {"custom_max_retention_hours": 0} clears it (an empty PATCH is otherwise
// rejected with "Nothing to update").
var catalogForceSend = []string{"Comment", "CustomMaxRetentionHours"}

Expand Down
6 changes: 3 additions & 3 deletions bundle/statemgmt/direct_migration.go
Original file line number Diff line number Diff line change
Expand Up @@ -172,9 +172,9 @@ func recordAutoMigrateSource(b *bundle.Bundle, requiredEngine engine.EngineSetti
}

// checkPlanOnTempState opens the migrated state at tempStatePath in read mode,
// runs a full plan against it, and returns the plan (and a non-nil error if the
// plan fails). Individual planning errors are emitted as warnings with warnPrefix
// so they are visible without failing the deploy. The plan is run in an isolated
// runs a full plan against it, and returns the plan, or a non-nil error if the
// plan fails (the caller fails the migration on it). Diagnostics collected while planning
// are additionally logged as warnings with warnPrefix for visibility. The plan is run in an isolated
// context so its diagnostics do not affect the deploy's own error state. The
// returned plan lets the caller inspect the planned actions (e.g. reject a
// migration that would recreate a resource).
Expand Down
4 changes: 2 additions & 2 deletions bundle/statemgmt/state_push.go
Original file line number Diff line number Diff line change
Expand Up @@ -25,8 +25,8 @@ func PushResourcesState(ctx context.Context, b *bundle.Bundle) {

local, err := os.Open(localPath)
if errors.Is(err, fs.ErrNotExist) {
// The state file can be absent if terraform apply is skipped because
// there are no changes to apply in the plan.
// The state file can be absent if the deploy made no changes
// that needed to be recorded.
log.Debugf(ctx, "Local state file does not exist: %s", localPath)
return
}
Expand Down
2 changes: 1 addition & 1 deletion bundle/terraform_dabs_map/types.go
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
// Package terraform_dabs_map provides a precomputed bidirectional mapping between
// DABs resource fields and Terraform resource fields, generated by generate_test.go.
// DABs resource fields and Terraform resource fields, frozen in generated.go (a snapshot, since the generator was removed with the Terraform engine).
package terraform_dabs_map

import "github.com/databricks/cli/libs/structs/structpath"
Expand Down
6 changes: 3 additions & 3 deletions cmd/bundle/deployment/migrate.go
Original file line number Diff line number Diff line change
Expand Up @@ -73,8 +73,8 @@ to the workspace so that subsequent deploys of this bundle use direct deployment
cmd.RunE = func(cmd *cobra.Command, args []string) error {
extraArgsStr := getCommonArgs(cmd)

// Clear the engine env var so migrate always uses terraform engine to read existing state,
// regardless of what the user may have set in their environment.
// Clear the engine env var so the user's environment cannot affect how the existing
// Terraform state is read for migration.
cmd.SetContext(env.Set(cmd.Context(), engine.EnvVar, ""))

opts := utils.ProcessOptions{
Expand All @@ -93,7 +93,7 @@ to the workspace so that subsequent deploys of this bundle use direct deployment
if stateDesc.Lineage == "" {
cmdio.LogString(ctx, `Error: This command migrates the existing Terraform state file (terraform.tfstate) to a direct deployment state file (resources.json). However, no existing local or remote state was found.

To start using direct engine, set "engine: direct" under bundle in your databricks.yml or deploy with DATABRICKS_BUNDLE_ENGINE=direct env var set.`)
New bundles use the direct engine automatically, no migration is needed.`)
return root.ErrAlreadyPrinted
}

Expand Down
2 changes: 1 addition & 1 deletion docs/direct.md
Original file line number Diff line number Diff line change
Expand Up @@ -37,7 +37,7 @@ There are known issues, see https://github.com/databricks/cli/issues?q=state%3Ao
### Migrating the existing deployment

The direct engine uses its own state file, also JSON, but with a different schema from terraform state file.
When a bundle still has Terraform state (`.databricks/bundle/<target>/terraform/terraform.tfstate` or its copy in the workspace), the CLI converts it to direct state in memory before running the command. Commands that only read state (e.g. `bundle plan`, `bundle summary`) work on the converted state without writing it. `bundle deploy` and `bundle destroy` commit the migration: the direct state is written locally and to the workspace, and the Terraform state is kept as `terraform.tfstate.backup`.
When a bundle still has Terraform state (`.databricks/bundle/<target>/terraform/terraform.tfstate` or its copy in the workspace), the CLI converts it to direct state in memory before running the command. Commands that only read state (e.g. `bundle plan`, `bundle summary`) work on the converted state without writing it. `bundle deploy` and `bundle destroy` commit the migration: the direct state is written locally and to the workspace, and the Terraform state is retired: the workspace copy is backed up and the local `terraform.tfstate` is renamed to `terraform.tfstate.backup`. `bundle destroy` removes all local state for the target, including that backup.

If the automatic migration fails, the command fails with an error; please report it to dabs-feedback@databricks.com. To keep deploying with Terraform in the meantime, use Databricks CLI v1.19.x, the last version that ships the Terraform engine.

Expand Down
4 changes: 2 additions & 2 deletions libs/telemetry/protos/bundle_deploy.go
Original file line number Diff line number Diff line change
Expand Up @@ -115,9 +115,9 @@ type BundleDeployExperimental struct {
// BundleResourcesMetadata mirrors the universe proto. Per-resource-type counts
// and state-size metadata for one bundle deployment.
//
// Counts cover both engines. Sizes are direct-only: the direct engine stores each
// Counts cover all resources. Sizes are only reported for the direct engine: the direct engine stores each
// resource's state as a JSON blob in resources.json, so a size is len(state) and
// nothing is serialized at telemetry time. Terraform entries carry counts only.
// nothing is serialized at telemetry time. Entries from the removed Terraform engine (older CLI versions) carry counts only.
type BundleResourcesMetadata struct {
// Engine that ran the deploy: "direct" or "terraform".
StateEngine string `json:"state_engine,omitempty"`
Expand Down
Loading