Repository navigation
feat(bigquery-jdbc): read large results from the job stream - #15078
keshavdandeva wants to merge 3 commits into
Conversation
There was a problem hiding this comment.
Code Review
This pull request enables reading query results directly from a query job's stream when EnableQueryResultsArrowFormat is enabled and no destination table is set, bypassing the need for a read session. It refactors BigQueryStatement to support this flow, including extracting helper methods for result set creation and exception handling, and adds comprehensive unit and integration tests. The reviewer suggests improving the extracted arrowResultSetException method by returning the exception directly if it is already an instance of BigQueryJdbcException, thereby avoiding redundant wrapping and preserving original exception properties.
| if (useJobStream(results)) { | ||
| LOG.info("Using the job stream to read the data."); | ||
| resultSet = processJobStreamResultSet(results, job); | ||
| this.currentExecutionBuilder.setQueryApiType(QueryApiType.QUERY_API_TYPE_READ_API); |
There was a problem hiding this comment.
Can we move this to before processJobStreamResultSet? So we still get telemetry even if processJobStreamResultSet throws.
Also, should this still be READ API?
| + " threshold: %d.", | ||
| results.hasNextPage(), rows, rowsInPage, threshold); | ||
| if (!results.hasNextPage() || rows < threshold) { | ||
| return false; |
b/582676785
With
EnableQueryResultsArrowFormat=true, a large SELECT result is read as Arrow from the query job's_defaultstream, using the reader from #15048. With the property off, results are read as before. The one visible change is the error message described under Changes.Changes
BigQueryStatement.processQueryResponsegets one new branch. It coversjobs.query, long-running jobs and each child job of a script.useJobStream(TableResult)holds the rule. A result is streamed from row 0 when noLargeResultTableorLargeResultDatasetis set, it has a next page, and it returns at leastHighThroughputMinTableSizerows (default 10,000) and more rows than its first page holds. Rows are counted aftersetMaxRows. Any other result is read as JSON. With a destination table set, today's High-Throughput API rules apply.cancel()could set it to null.new BigQueryJdbcException(ex). Its message is no longer repeated, and aBigQueryJdbcExceptionis passed on as a copy with its SQLState instead of being nested in a second one. This also changes the error message of the existing High-Throughput path, which repeated the message before.EnableQueryResultsArrowFormatandEnableHighThroughputAPIsay how the two combine.Not in this PR
SQLException.maxRowsis reached (b/586660667).cancel()that lands before the statement stores the result set can be lost: it returns normally and the statement still gets the result set. This is not new, and it is the same on the JSON, read-session and job-stream paths.Decisions to check
HighThroughputMinTableSize, not a new property.MaxResults.disable_tpc.