[SPARK-57199][SQL] Extract the aggregate out-of-memory error into a QueryExecutionErrors factory#56256
Closed
gengliangwang wants to merge 1 commit into
Closed
[SPARK-57199][SQL] Extract the aggregate out-of-memory error into a QueryExecutionErrors factory#56256gengliangwang wants to merge 1 commit into
gengliangwang wants to merge 1 commit into
Conversation
…ueryExecutionErrors factory
The AGGREGATE_OUT_OF_MEMORY error is constructed inline in HashAggregateExec
codegen (emitted into every generated aggregate class) and in
TungstenAggregationIterator. Add a QueryExecutionErrors.aggregateOutOfMemoryError()
factory and route both call sites through it.
Across the TPC-DS queries the inline
`throw new SparkOutOfMemoryError("AGGREGATE_OUT_OF_MEMORY", new java.util.HashMap());`
appears 445 times (the most-repeated throw in the generated code); the factory
call shrinks each generated class and moves the error-class string + empty
message map into one compiled method. Behavior is unchanged.
Generated-by: Claude Code (Opus 4.8)
LuciferYang
approved these changes
Jun 3, 2026
gengliangwang
added a commit
that referenced
this pull request
Jun 3, 2026
…ueryExecutionErrors factory
### What changes were proposed in this pull request?
The aggregate out-of-memory error (`AGGREGATE_OUT_OF_MEMORY`) is constructed inline in two places:
- `HashAggregateExec`, whose whole-stage codegen emits `throw new <SparkOutOfMemoryError>("AGGREGATE_OUT_OF_MEMORY", new java.util.HashMap());` into every generated aggregate class.
- `TungstenAggregationIterator` (the interpreted fallback), which throws the same `new SparkOutOfMemoryError(...)` and needs a `// scalastyle:off throwerror` suppression.
This PR adds a `QueryExecutionErrors.aggregateOutOfMemoryError()` factory (next to the existing `cannotAcquireMemory*` OOM factories) and routes both call sites through it. In the codegen path the emitted Java becomes `throw QueryExecutionErrors.aggregateOutOfMemoryError();`.
### Why are the changes needed?
Sub-task of SPARK-56908 (reduce generated Java size in whole-stage codegen). Dumping the whole-stage codegen of the TPC-DS queries shows the inline `throw new org.apache.spark.memory.SparkOutOfMemoryError("AGGREGATE_OUT_OF_MEMORY", new java.util.HashMap());` line **445 times** across 142 of 150 generated classes -- it is the single most-repeated `throw` in the corpus. Replacing it with a factory call shrinks each generated aggregate class and moves the error-class string and the empty message-parameter map out of every generated class's constant pool into one compiled method. It also consolidates the error construction shared with the interpreted path and removes the `throwerror` scalastyle suppression there.
### Does this PR introduce _any_ user-facing change?
No. The same `AGGREGATE_OUT_OF_MEMORY` error with the same (empty) message parameters is thrown; only where it is constructed changes.
### How was this patch tested?
This is a behavior-preserving refactor, covered by the existing aggregate suites (e.g. `DataFrameAggregateSuite`, 163 tests, pass). The change was additionally verified by re-dumping the TPC-DS whole-stage codegen: all 445 inline throws are now `QueryExecutionErrors.aggregateOutOfMemoryError()` calls, and every generated subtree still compiles (the Janino default imports already make `QueryExecutionErrors` available unqualified, as used by other generated error calls such as `divideByZeroError`). This mirrors the sibling `DateTimeExpressionUtils` codegen extractions, which likewise relied on existing expression-suite coverage.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Opus 4.8)
Closes #56256 from gengliangwang/spark-agg-oom-factory.
Authored-by: Gengliang Wang <gengliang@apache.org>
Signed-off-by: Gengliang Wang <gengliang@apache.org>
(cherry picked from commit 4f70b60)
Signed-off-by: Gengliang Wang <gengliang@apache.org>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
The aggregate out-of-memory error (
AGGREGATE_OUT_OF_MEMORY) is constructed inline in two places:HashAggregateExec, whose whole-stage codegen emitsthrow new <SparkOutOfMemoryError>("AGGREGATE_OUT_OF_MEMORY", new java.util.HashMap());into every generated aggregate class.TungstenAggregationIterator(the interpreted fallback), which throws the samenew SparkOutOfMemoryError(...)and needs a// scalastyle:off throwerrorsuppression.This PR adds a
QueryExecutionErrors.aggregateOutOfMemoryError()factory (next to the existingcannotAcquireMemory*OOM factories) and routes both call sites through it. In the codegen path the emitted Java becomesthrow QueryExecutionErrors.aggregateOutOfMemoryError();.Why are the changes needed?
Sub-task of SPARK-56908 (reduce generated Java size in whole-stage codegen). Dumping the whole-stage codegen of the TPC-DS queries shows the inline
throw new org.apache.spark.memory.SparkOutOfMemoryError("AGGREGATE_OUT_OF_MEMORY", new java.util.HashMap());line 445 times across 142 of 150 generated classes -- it is the single most-repeatedthrowin the corpus. Replacing it with a factory call shrinks each generated aggregate class and moves the error-class string and the empty message-parameter map out of every generated class's constant pool into one compiled method. It also consolidates the error construction shared with the interpreted path and removes thethrowerrorscalastyle suppression there.Does this PR introduce any user-facing change?
No. The same
AGGREGATE_OUT_OF_MEMORYerror with the same (empty) message parameters is thrown; only where it is constructed changes.How was this patch tested?
This is a behavior-preserving refactor, covered by the existing aggregate suites (e.g.
DataFrameAggregateSuite, 163 tests, pass). The change was additionally verified by re-dumping the TPC-DS whole-stage codegen: all 445 inline throws are nowQueryExecutionErrors.aggregateOutOfMemoryError()calls, and every generated subtree still compiles (the Janino default imports already makeQueryExecutionErrorsavailable unqualified, as used by other generated error calls such asdivideByZeroError). This mirrors the siblingDateTimeExpressionUtilscodegen extractions, which likewise relied on existing expression-suite coverage.Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Opus 4.8)