Repository navigation
[Java] BigQueryIO: set table labels from dynamic destinations - #40843
Open
mxtymoshyk wants to merge 3 commits into
Open
mxtymoshyk wants to merge 3 commits into
mxtymoshyk wants to merge 3 commits into
Conversation
Add TableDestination.withLabels() and TableDestinationCoderV4. Streaming inserts and the Storage Write API set labels when they create the table. FILE_LOADS patches them after the load or copy job. withTableLabels() switches to(SerializableFunction) writes to the new coder. Fixes apache#26386.
Contributor
|
Checks are failing. Will not request review until checks are succeeding. If you'd like to override that behavior, comment |
Contributor
|
Assigning reviewers: R: @Abacn for label java. Note: If you would like to opt out of this review, comment Available commands:
The PR bot will only process comments in the main thread (not review comments). |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #26386.
TableDestinationgetswithLabels(Map)andgetLabels(). BigQueryIO applies those labels to the destination table:The labels have to survive encoding of the destination. When the destination type is
TableDestination, the V1 to V3 coders drop them, so this addsTableDestinationCoderV4. It is opt-in so existing streaming pipelines can still be updated in place:to(SerializableFunction), callwithTableLabels().to(DynamicDestinations), returnTableDestinationCoderV4fromgetDestinationCoder().Moving a running streaming pipeline to V4 changes how destinations are encoded, so it needs a drain, the same as turning on
withClustering().If a destination has labels and its coder would drop them, the write fails with a message that says which option to set.
withTableLabels()on a single-table write fails when the pipeline is built.Two other places dropped labels and are fixed here: the wrapper that applies
withTimePartitioning()andwithClustering()to dynamic destinations, and the temp-table path that reads an existing table's partitioning.Testing: new tests in BigQueryIOWriteTest cover FILE_LOADS, streaming inserts, temp tables on a table that already has labels, and the error cases. TableDestinationCoderV4Test covers round trips and deterministic encoding. Both pass locally. I have not run this against real BigQuery, and the Storage Write API path has no direct test.