Skip to content

[Java] BigQueryIO: set table labels from dynamic destinations - #40843

Open
mxtymoshyk wants to merge 3 commits into
apache:masterfrom
mxtymoshyk:feat/bq-table-labels
Open

mxtymoshyk wants to merge 3 commits into
apache:masterfrom
mxtymoshyk:feat/bq-table-labels

Conversation

@mxtymoshyk

Copy link
Copy Markdown
Contributor

Fixes #26386.

TableDestination gets withLabels(Map) and getLabels(). BigQueryIO applies those labels to the destination table:

  • STREAMING_INSERTS and STORAGE_WRITE_API set them when they create the table.
  • FILE_LOADS patches them onto the table after the load job, or after the copy job when the write goes through temp tables. Copy jobs have no label setting, so a patch is the only option. The patch merges with labels already on the table.

The labels have to survive encoding of the destination. When the destination type is TableDestination, the V1 to V3 coders drop them, so this adds TableDestinationCoderV4. It is opt-in so existing streaming pipelines can still be updated in place:

  • With to(SerializableFunction), call withTableLabels().
  • With to(DynamicDestinations), return TableDestinationCoderV4 from getDestinationCoder().

Moving a running streaming pipeline to V4 changes how destinations are encoded, so it needs a drain, the same as turning on withClustering().

If a destination has labels and its coder would drop them, the write fails with a message that says which option to set. withTableLabels() on a single-table write fails when the pipeline is built.

Two other places dropped labels and are fixed here: the wrapper that applies withTimePartitioning() and withClustering() to dynamic destinations, and the temp-table path that reads an existing table's partitioning.

Testing: new tests in BigQueryIOWriteTest cover FILE_LOADS, streaming inserts, temp tables on a table that already has labels, and the error cases. TableDestinationCoderV4Test covers round trips and deterministic encoding. Both pass locally. I have not run this against real BigQuery, and the Storage Write API path has no direct test.

Add TableDestination.withLabels() and TableDestinationCoderV4. Streaming
inserts and the Storage Write API set labels when they create the table.
FILE_LOADS patches them after the load or copy job. withTableLabels()
switches to(SerializableFunction) writes to the new coder.

Fixes apache#26386.
@github-actions github-actions Bot added java io gcp size/l L: 250-999 lines changed labels Oct 27, 2026
@github-actions

Copy link
Copy Markdown
Contributor

Checks are failing. Will not request review until checks are succeeding. If you'd like to override that behavior, comment assign set of reviewers

@github-actions

Copy link
Copy Markdown
Contributor

Assigning reviewers:

R: @Abacn for label java.
Backup reviewer: kennknowles (if the main reviewer is unresponsive, you can escalate to them by tagging their username)

Note: If you would like to opt out of this review, comment assign to next reviewer.

Available commands:

  • stop reviewer notifications - opt out of the automated review tooling
  • remind me after tests pass - tag the comment author after tests pass
  • waiting on author - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)

The PR bot will only process comments in the main thread (not review comments).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Feature Request]: Add ability to pass labels to BQ table in DynamicDestinations

1 participant