Skip to content

[FLINK-39219] [table] Support lineage extraction for DataStreamscan/sink providers - #27727

Open
jsinghkukreja wants to merge 3 commits into
apache:masterfrom
jsinghkukreja:support-lineage-for-datastream-providers
Open

jsinghkukreja wants to merge 3 commits into
apache:masterfrom
jsinghkukreja:support-lineage-for-datastream-providers

Conversation

@jsinghkukreja

@jsinghkukreja jsinghkukreja commented Mar 3, 2026 •

Copy link
Copy Markdown

What is the purpose of the change

This pull request adds lineage vertex extraction support for DataStreamScanProvider and DataStreamSinkProvider.

Brief change log

  • This pull request adds lineage vertex extraction support for DataStreamScanProvider and DataStreamSinkProvider in the table planner.
  • Previously, only Source, SinkFunction, and SinkV2 providers had their LineageVertex extracted for FLIP-314 lineage events.
  • Connectors that use DataStream providers (e.g. Apache Paimon) were silently missing lineage dataset facets. This ensures all scan/sink provider types are treated consistently for lineage reporting.
  • Specifically was trying to solve this issue for reference: [Feature] Support native Flink 2.0 job lineage (FLIP-314) for Paimon tables paimon#7306

Verifying this change

This change added tests and can be verified as follows:

  • Added unit tests for the changes done in flink-table/flink-table-planner/src/test/java/org/apache/flink/table/planner/lineage/TableLineageUtilsTest.java.
  • Verified manually by running a flink cluster that the changes made are running as expected.

Does this pull request potentially affect one of the following parts:

  • Dependencies (does it add or upgrade a dependency): (no)
  • The public API, i.e., is any changed class annotated with @Public(Evolving): ( no)
  • The serializers: (no)
  • The runtime per-record code paths (performance sensitive): (no)
  • Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (no)
  • The S3 file system connector: (no)

Documentation

  • Does this pull request introduce a new feature? (no)

@flinkbot

flinkbot commented Mar 3, 2026 •

Copy link
Copy Markdown
Collaborator

CI report:

Bot commands The @flinkbot bot supports the following commands:
  • @flinkbot run azure re-run the last Azure build

@jsinghkukreja
jsinghkukreja force-pushed the support-lineage-for-datastream-providers branch from 9fef5c9 to 16bda1f Compare March 3, 2026 04:55
@jsinghkukreja

Copy link
Copy Markdown
Author

I don't have access to the JIRA board I have requested for access in the mean time can anyone please help creating a ticket for this PR?

@jsinghkukreja jsinghkukreja changed the title [FLINK-XXXX] [table] Support lineage extraction for DataStreamscan/sink providers [FLINK-39219] [table] Support lineage extraction for DataStreamscan/sink providers Mar 6, 2026
@jsinghkukreja
jsinghkukreja force-pushed the support-lineage-for-datastream-providers branch from 16bda1f to 81c4b93 Compare March 10, 2026 23:41
@github-actions github-actions Bot added the community-reviewed PR has been reviewed by the community. label Mar 11, 2026

@HuangZhenQiu HuangZhenQiu left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. @rmetzger Would you please further review?

@jsinghkukreja

Copy link
Copy Markdown
Author

@rmetzger Friendly bump can I please get a review on this PR?

@jsinghkukreja

jsinghkukreja commented Jun 15, 2026 •

Copy link
Copy Markdown
Author

cc: @rmetzger , @MartijnVisser , @fhueske friendly bump can I please have a review on this PR, thanks.

@tub

tub commented Jul 20, 2026

Copy link
Copy Markdown

It'd be awesome if this were merged so we could get lineage data for our Paimon workloads! :)

@github-actions

Copy link
Copy Markdown

This PR is being marked as stale since it has not had any activity in the last 90 days.
If you would like to keep this PR alive, please leave a comment asking for a review.
If the PR has merge conflicts, update it with the latest from the base branch.

If you are having difficulty finding a reviewer, please reach out to the
community, contact details can be found here: https://flink.apache.org/what-is-flink/community/

If this PR is no longer valid or desired, please feel free to close it.
If no activity occurs in the next 30 days, it will be automatically closed.

@github-actions github-actions Bot added the stale label Oct 19, 2026
@github-actions

Copy link
Copy Markdown

This PR has been closed since it has not had any activity in 120 days.
If you feel like this was a mistake, or you would like to continue working on it,
please feel free to re-open the PR and ask for a review.

@jsinghkukreja

jsinghkukreja commented Nov 19, 2026 •

Copy link
Copy Markdown
Author

Could a maintainer please reopen this PR? I am unable to reopen it myself as it was closed by the stale bot and GitHub does not provide the 'Reopen' button for non-maintainers in this case. @HuangZhenQiu/@rmetzger

@MartijnVisser MartijnVisser reopened this Nov 19, 2026
@MartijnVisser MartijnVisser self-assigned this Nov 19, 2026

@MartijnVisser MartijnVisser left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reopened, Paimon already documents this as the Flink side it needs. The fromDataStream comment needs a fix before merge. Please also rebase, TransformationsTest conflicts with master, and add a sentence to the lineage docs that DataStream providers can implement LineageVertexProvider too.

if (sourceTransform instanceof TransformationWithLineage) {
((TransformationWithLineage<RowData>) sourceTransform)
.setLineageVertex(sourceLineageVertex);
} else {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This also runs for TransformationScanProvider, so fromDataStream over a source with its own lineage now reports an anonymous table with an empty namespace instead of the connector's dataset. I think this should only run for DataStreamScanProvider.

&& predecessor instanceof TransformationWithLineage) {
((TransformationWithLineage<?>) predecessor)
.setLineageVertex(sourceLineageVertex);
break;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

With a union only the first source gets the table vertex. Might make sense to require exactly one source.

.getTransformation();
metadata.fill(sourceTransform);
} else if (provider instanceof DataStreamScanProvider) {
lineageVertex = TableLineageUtils.extractLineageDataset(provider);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Providers that aren't a LineageVertexProvider themselves, like Kafka's, still lose the connector's namespace. Falling back to the vertex already on the source would fix that. Here or in a follow-up?

}

@Test
void testExtractLineageDatasetFromDataStreamScanProvider() {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These four also pass on master, since extractLineageDataset didn't change, so they can go.

@Override
public DataStream<RowData> produceDataStream(
ProviderContext providerContext, StreamExecutionEnvironment execEnv) {
// Single-operator pipeline: returns the SourceTransformation directly

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The map makes this a multi-operator pipeline as well, so a provider that returns the source directly isn't covered.

Restrict upstream traversal to DataStreamScanProvider pipelines with one
distinct source, preserving existing lineage for fromDataStream conversions.
Fall back to source-provided lineage when the provider exposes none.

Add regression coverage, remove redundant tests, and document
LineageVertexProvider support for DataStream providers.
@jsinghkukreja
jsinghkukreja force-pushed the support-lineage-for-datastream-providers branch from f00e6e1 to 409281f Compare November 20, 2026 07:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

community-reviewed PR has been reviewed by the community.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants