Treat a 404 from BigQuery table and dataset deletes as success - #39934
Open
mxtymoshyk wants to merge 1 commit into
Open
Treat a 404 from BigQuery table and dataset deletes as success#39934mxtymoshyk wants to merge 1 commit into
mxtymoshyk wants to merge 1 commit into
Conversation
BigQueryIO deletes the temporary tables and datasets it creates. A delete can succeed at BigQuery and still have its work item fail to commit afterwards; the runner then replays the work item and the replayed delete gets a 404 because the first attempt already removed the resource. deleteTable and deleteDataset passed ALWAYS_RETRY, so that 404 was retried MAX_RPC_RETRIES times and then thrown. In a streaming job the work item retries forever, which stalls a drain. Both now use DONT_RETRY_NOT_FOUND and swallow an item-not-found error, matching how getTable already handles a 404. Every other status keeps its existing retry and failure behaviour.
Contributor
|
Assigning reviewers: R: @kennknowles for label java. Note: If you would like to opt out of this review, comment Available commands:
The PR bot will only process comments in the main thread (not review comments). |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
BigQueryIO creates temporary tables and datasets and deletes them when it is done. A delete can succeed at BigQuery and still have its work item fail to commit afterwards. The runner then replays that work item, and the replayed delete gets a 404 because the first attempt already removed the resource.
DatasetServiceImpl.deleteTableanddeleteDatasetpassedALWAYS_RETRY, so that 404 was retriedMAX_RPC_RETRIES(9) times with exponential backoff and then rethrown as anIOException. In a streaming job the work item is retried forever and the job cannot drain.Both methods now pass
DONT_RETRY_NOT_FOUNDand swallow an item-not-found error, since "the resource is gone" is the outcome the caller asked for. This mirrors howgetTablein the same class already handles a 404. Every other status code keeps its existing retry count and failure behaviour.fixes #24997
Notes for reviewers
dryRunQueryandpatchTableDescriptionstill useALWAYS_RETRY, and nogetX/create/patchpath changed.ApiErrorExtractor.itemNotFoundwalksgetCause(), so it still recognises the 404 afterexecuteWithRetrieswraps it in a newIOException-- the same reason the existinggetTablecatch block works.Verifyin the shared mock request. They assert both "did not throw" and "did not retry".:sdks:java:io:google-cloud-platform:BigQueryServicesImplTestpasses 50/50 with the change. Reverting only theBigQueryServicesImplchange makes both new tests fail with aVerifyExceptionfrom the retried request, confirming the tests cover the fix.