Skip the apply_mutators phase when no mutators are configured - #6294
Open
bjornkpu wants to merge 3 commits into
Open
Skip the apply_mutators phase when no mutators are configured#6294bjornkpu wants to merge 3 commits into
bjornkpu wants to merge 3 commits into
Conversation
getOpts already receives the phase but only decides `enabled` from whether
Python configuration is present, so `apply_mutators` starts a subprocess even
when no mutator functions are declared. That subprocess loads every resource
through the databricks-bundles dataclasses and returns them unchanged, so the
work is wasted for bundles that use `python.resources` without
`python.mutators`.
It is also not harmless. Any resource the generated dataclasses cannot
construct fails the whole phase, and therefore the whole command, even though
the resource is defined in YAML and the user asked for nothing to be mutated. A
SharePoint file-ingestion pipeline hits this because TableSpec.source_table is
required while a file-ingestion object has no source table:
TypeError: TableSpec.__init__() missing 1 required keyword-only argument:
'source_table'
Configuration errors this phase would report are unaffected, because
load_resources runs first and derives its options the same way.
Contributor
Approval status: pending
|
Contributor
|
An authorized user can trigger integration tests manually by following the instructions below: Trigger: Inputs:
Checks will be approved automatically on success. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Changes
Skip the
apply_mutatorsphase when no mutator functions are configured.Why
getOptsreceives the phase but decidesenabledonly from whether Python configuration is present. Soapply_mutatorsstarts a subprocess even when no mutator functions are declared. That subprocess loads every resource through thedatabricks-bundlesdataclasses and returns them unchanged.It is not only wasted work. Any resource the dataclasses cannot construct fails the phase, and therefore the command, even though the resource is defined in YAML and nothing was asked to be mutated. A SharePoint file-ingestion pipeline hits this:
source_tableis required onTableSpecwhile a file-ingestion object has no source table. That is a separate problem, filed as #6295. This PR does not fix it. It stops such a bundle from failing in a phase it does not use.Configuration errors this phase would report are unaffected.
load_resourcesruns first and derives its options the same way.Tests
Three unit tests on
getOpts:apply_mutatorswith no mutators, withpython.mutators, and withexperimental.python.mutators.No acceptance test changes. Nothing under
acceptance/observes this phase, and the output is identical either way, since the subprocess returned its input unchanged.