[FLINK-39847][table] Support in-place conversion of a table to a materialized table by raminqaf · Pull Request #28287 · apache/flink

raminqaf · 2026-06-01T14:40:59Z

What is the purpose of the change

This is the implementation of the FLIP-578: In-place Table to Materialized Table conversion

Introduce the building blocks for in-place conversion of a regular table to a materialized table use the already existing CoA command.

Brief change log

Add the cluster-level table.materialized-table.conversion-from-table.enabled option (default false) that gates the conversion. It is read from the root configuration, so a session-level SET has no effect.
Add Catalog#convertTableToMaterializedTable, which swaps an existing regular table's catalog entry for a materialized table in place while preserving its identity and storage. The default throws UnsupportedOperationException; GenericInMemoryCatalog overrides it with a table-kind check. CatalogManager resolves both tables, delegates to the catalog, and fires an AlterTableEvent. Launching the refresh job is left to the executor, not the catalog.
handleCreateOrAlter now dispatches on the kind of the existing object: a materialized table is altered as before, a view is rejected, and a regular table is converted in place to a materialized table when table.materialized-table.conversion-from-table.enabled is set on the cluster (otherwise the statement is rejected as before). The conversion carries over the source table's watermark and primary key when the DDL omits them, rejecting a source with more than one watermark. It emits the structured TableChanges (columns, constraint, watermark, definition query, options, distribution, start mode) by reusing the shared change-building helpers, and returns a ConvertTableToMaterializedTableOperation.

Verifying this change

Added test in SqlNodeToOperationConvertTableToMaterializedTableTest
Added ITs in MaterializedTableStatementITCase

Does this pull request potentially affect one of the following parts:

Dependencies (does it add or upgrade a dependency): no
The public API, i.e., is any changed class annotated with @Public(Evolving): yes
The serializers: no
The runtime per-record code paths (performance sensitive): no
Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: no
The S3 file system connector: no

Documentation

Does this pull request introduce a new feature? yes
If yes, how is the feature documented? docs/JavaDocs

Was generative AI tooling used to co-author this PR?

Yes (please specify the tool below)

Generated-by: Opus 4.8

flinkbot · 2026-06-01T14:50:05Z

CI report:

b16909a Azure: SUCCESS

Bot commands

The @flinkbot bot supports the following commands:

@flinkbot run azure re-run the last Azure build

AHeise

First pass.

AHeise

LAGTM. We need to figure out if we need a special conversion element.

snuyanzin · 2026-06-11T13:31:57Z

can you elaborate how do these match (from commit message)

The conversion carries over the source table's watermark and primary key when the DDL omits them, rejecting a source with more than one watermark.

and the test in PR sourceWatermarkAndPrimaryKeyAreNotInherited?

snuyanzin · 2026-06-11T13:38:14Z

also another commit message

On a refresh-job-launch failure the conversion makes a best-effort rollback: it drops the materialized table and recreates the original regular table from the operation's original table.

vs what I see in doc

If the refresh job (continuous mode) or refresh workflow (full mode) cannot be started, the conversion is not rolled back: the table is left as a materialized table in SUSPENDED status. Fix the underlying issue and RESUME it.

raminqaf · 2026-06-14T13:57:23Z

can you elaborate how do these match (from commit message)

The conversion carries over the source table's watermark and primary key when the DDL omits them, rejecting a source with more than one watermark.

and the test in PR sourceWatermarkAndPrimaryKeyAreNotInherited?

Updated the commit message to reflect the logic of the code

twalthr

Thanks @raminqaf. I left some last comments. Should be good in the next iteration.

twalthr

LGTM, thanks @raminqaf!

…aterialized table conversion Introduce the building blocks for in-place conversion of a regular table to a materialized table. Neither part is wired into SQL execution on its own. Add the cluster-level table.materialized-table.conversion-from-table.enabled option (default false) that gates the conversion. It is read from the root configuration, so a session-level SET has no effect. Add Catalog#convertTableToMaterializedTable, which swaps an existing regular table's catalog entry for a materialized table in place while preserving its identity and storage. The default throws UnsupportedOperationException; GenericInMemoryCatalog overrides it with a table-kind check. CatalogManager resolves both tables, delegates to the catalog, and fires an AlterTableEvent. Launching the refresh job is left to the executor, not the catalog.

…CREATE OR ALTER handleCreateOrAlter now dispatches on the kind of the existing object: a materialized table is altered as before, a view is rejected, and a regular table is converted in place to a materialized table when table.materialized-table.conversion-from-table.enabled is set on the cluster (otherwise the statement is rejected as before). The conversion carries over the source table's watermark and primary key when the DDL defines them, rejecting a source with more than one watermark. It emits the structured TableChanges (columns, constraint, watermark, definition query, options, distribution, start mode) by reusing the shared change-building helpers, and returns a ConvertTableToMaterializedTableOperation.

…materialized table The SQL gateway's MaterializedTableManager now handles ConvertTableToMaterializedTableOperation: it swaps the catalog entry and then launches the refresh job (a continuous streaming job or a full-mode workflow) and persists the refresh handler, mirroring CREATE MATERIALIZED TABLE. Without this the gateway rejected the operation as unsupported, so conversion swapped the catalog entry but never started refreshing. On a refresh-job-launch failure the conversion makes a best-effort rollback: it drops the materialized table and recreates the original regular table from the operation's original table. Unlike CREATE there is no clean reverse path (the catalog only converts table to materialized table, and alterTable rejects kind changes), so this restores the user's prior state rather than dropping the table outright. TestFileSystemCatalog gains the conversion override and the materialized table ITCase base enables the cluster-level conversion flag so the path can be exercised end to end.

…version

AHeise reviewed Jun 2, 2026

View reviewed changes

raminqaf changed the title ~~[FLINK-XXXXX][table] Add config option and Catalog API for table to materialized table conversion~~ [FLINK-39847][table] Support in-place conversion of a table to a materialized table Jun 4, 2026

raminqaf force-pushed the flip-578-table-to-mt-conversion branch 7 times, most recently from 8e9f442 to 62aa1b4 Compare June 5, 2026 08:44

raminqaf marked this pull request as ready for review June 5, 2026 08:54

raminqaf force-pushed the flip-578-table-to-mt-conversion branch from 62aa1b4 to f8a9683 Compare June 5, 2026 08:58

AHeise reviewed Jun 5, 2026

View reviewed changes

Comment thread ...-table/flink-table-api-java/src/main/java/org/apache/flink/table/catalog/CatalogManager.java

raminqaf force-pushed the flip-578-table-to-mt-conversion branch from f8a9683 to d422760 Compare June 5, 2026 10:05