Repository navigation
[FLINK-39219] [table] Support lineage extraction for DataStreamscan/sink providers - #27727
jsinghkukreja wants to merge 3 commits into
Conversation
9fef5c9 to
16bda1f
Compare
|
I don't have access to the JIRA board I have requested for access in the mean time can anyone please help creating a ticket for this PR? |
16bda1f to
81c4b93
Compare
HuangZhenQiu
left a comment
There was a problem hiding this comment.
LGTM. @rmetzger Would you please further review?
|
@rmetzger Friendly bump can I please get a review on this PR? |
|
cc: @rmetzger , @MartijnVisser , @fhueske friendly bump can I please have a review on this PR, thanks. |
|
It'd be awesome if this were merged so we could get lineage data for our Paimon workloads! :) |
|
This PR is being marked as stale since it has not had any activity in the last 90 days. If you are having difficulty finding a reviewer, please reach out to the If this PR is no longer valid or desired, please feel free to close it. |
|
This PR has been closed since it has not had any activity in 120 days. |
|
Could a maintainer please reopen this PR? I am unable to reopen it myself as it was closed by the stale bot and GitHub does not provide the 'Reopen' button for non-maintainers in this case. @HuangZhenQiu/@rmetzger |
MartijnVisser
left a comment
There was a problem hiding this comment.
Reopened, Paimon already documents this as the Flink side it needs. The fromDataStream comment needs a fix before merge. Please also rebase, TransformationsTest conflicts with master, and add a sentence to the lineage docs that DataStream providers can implement LineageVertexProvider too.
| if (sourceTransform instanceof TransformationWithLineage) { | ||
| ((TransformationWithLineage<RowData>) sourceTransform) | ||
| .setLineageVertex(sourceLineageVertex); | ||
| } else { |
There was a problem hiding this comment.
This also runs for TransformationScanProvider, so fromDataStream over a source with its own lineage now reports an anonymous table with an empty namespace instead of the connector's dataset. I think this should only run for DataStreamScanProvider.
| && predecessor instanceof TransformationWithLineage) { | ||
| ((TransformationWithLineage<?>) predecessor) | ||
| .setLineageVertex(sourceLineageVertex); | ||
| break; |
There was a problem hiding this comment.
With a union only the first source gets the table vertex. Might make sense to require exactly one source.
| .getTransformation(); | ||
| metadata.fill(sourceTransform); | ||
| } else if (provider instanceof DataStreamScanProvider) { | ||
| lineageVertex = TableLineageUtils.extractLineageDataset(provider); |
There was a problem hiding this comment.
Providers that aren't a LineageVertexProvider themselves, like Kafka's, still lose the connector's namespace. Falling back to the vertex already on the source would fix that. Here or in a follow-up?
| } | ||
|
|
||
| @Test | ||
| void testExtractLineageDatasetFromDataStreamScanProvider() { |
There was a problem hiding this comment.
These four also pass on master, since extractLineageDataset didn't change, so they can go.
| @Override | ||
| public DataStream<RowData> produceDataStream( | ||
| ProviderContext providerContext, StreamExecutionEnvironment execEnv) { | ||
| // Single-operator pipeline: returns the SourceTransformation directly |
There was a problem hiding this comment.
The map makes this a multi-operator pipeline as well, so a provider that returns the source directly isn't covered.
Restrict upstream traversal to DataStreamScanProvider pipelines with one distinct source, preserving existing lineage for fromDataStream conversions. Fall back to source-provided lineage when the provider exposes none. Add regression coverage, remove redundant tests, and document LineageVertexProvider support for DataStream providers.
f00e6e1 to
409281f
Compare
What is the purpose of the change
This pull request adds lineage vertex extraction support for DataStreamScanProvider and DataStreamSinkProvider.
Brief change log
Verifying this change
This change added tests and can be verified as follows:
Does this pull request potentially affect one of the following parts:
@Public(Evolving): ( no)Documentation