feat(bigquery-jdbc): support connection-scoped executor isolation and dynamic queue warnings#13413
feat(bigquery-jdbc): support connection-scoped executor isolation and dynamic queue warnings#13413Neenu1995 wants to merge 5 commits into
Conversation
… dynamic queue warnings
There was a problem hiding this comment.
Code Review
This pull request introduces a new connection property QueryExecutionThreadCount and an associated queryExecutor thread pool to handle background query execution. It also adds thread pool queue saturation monitoring with warning logs, configures pool threads as daemon threads, and includes comprehensive tests. The review feedback correctly points out a potential resource leak and sequential delay in the shutdown logic of BigQueryConnection when terminating the executors. If an interruption occurs during the termination of the metadata executor, the query executor's shutdown is skipped. A robust parallel shutdown approach with proper exception handling was suggested to resolve this issue.
|
/gemini review |
There was a problem hiding this comment.
Code Review
This pull request introduces a dedicated background query execution thread pool (queryExecutor) alongside the existing metadata fetch thread pool, including configurable thread counts, daemon thread configuration, and queue saturation monitoring with hysteresis warnings. The review feedback suggests optimizing the shutdown sequence in BigQueryConnection to avoid unnecessary blocking if an interrupt occurs, optimizing the warning flag reset in BigQueryJdbcMdc to prevent cache line bouncing, and using nested try-finally blocks in tests to ensure proper resource cleanup.
| } | ||
| this.openStatements.clear(); | ||
|
|
||
| if (this.metadataExecutor != null) { |
There was a problem hiding this comment.
I think we should update this closeImpl() to call all shutdown ops first & do await later, right now it awaits each shutdown individually.
We could force shutdownNow in Interrupted exception handler that already exists & we need to do it for all (including read/write clients), no need to handle them independently (I'd assume shutdownNow is no-op if was already successful)
Also, if Interrupted is thrown during bigQueryWriteClient.awaitTermination, openStatements is not properly cleaned up.
b/519201498
This PR implements connection-scoped thread pool isolation and dynamic saturation monitoring for the BigQuery JDBC driver.
queryExecutorandmetadataExecutorinstances per connection, properly cleaning them up on connection close.max(10, corePoolSize * 5)and resets only when it recovers belowmax(4, corePoolSize * 2).BigQueryJdbcMdcTestto inherit fromBigQueryJdbcLoggingBaseTest, and adds unit tests to verify lifecycle, saturation logging, and hysteresis behavior.