[hotfix][cdc-composer] Fix "URI is not hierarchical" when resolving factory JAR path - #4542
Open
wangmingzhou1986 wants to merge 3 commits into
Open
wangmingzhou1986 wants to merge 3 commits into
wangmingzhou1986 wants to merge 3 commits into
Conversation
…esolving factory JAR path
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What is the purpose of the change
FactoryDiscoveryUtils#getJarPathByIdentifierresolves the code source location of aconnector
Factoryto a local path in order to decide whether the containing JAR has to beshipped to the TaskManagers. It does so with
Paths.get(url.toURI()), which requires ahierarchical URI.
A code source location is not always hierarchical. When a pipeline runs in application mode
and Flink picks up the JARs under
usrlib, the user code class loader is built with relativeURLs, e.g.
file:usrlib/flink-cdc-pipeline-connector-mysql-3.6.0-1.20.jar. Such a URL has noauthority and its path does not start with a slash, so the URI is opaque and
Paths.get(URI)throws:which is wrapped into
RuntimeException: Failed to search JAR by factory identifier "mysql"and fails the job before it is ever submitted.
How it was observed
Running a CDC YAML pipeline as a Flink Kubernetes Operator
FlinkDeployment, with the CDCconnector JARs placed under
/opt/flink/usrlib. Printing the code source of each class frominside the JobManager shows the difference clearly:
CodeSourcelocationURI#isOpaquejarURI: local:///opt/flink/usrlib/...)file:/opt/flink/usrlib/entry.jarfalseMySqlDataSourceFactory(picked up by theusrlibscan)file:usrlib/flink-cdc-pipeline-connector-mysql-3.6.0-1.20.jartruePaimonDataSinkFactoryfile:usrlib/flink-cdc-pipeline-connector-paimon-3.6.0-1.20.jartrueOnly the absoluteness of the URL differs; the packaging is identical. The existing
local:///opt/flink/usrlib/short-circuit does not match these URLs, so execution reachesPaths.getand throws.Brief change log
resolveLocalPath(URL)helper.directory, and returns an absolute path in both cases.
Optional.empty()(i.e. do not attempt to upload the JAR) instead of throwing.no longer receive a working-directory-relative URL.
Behaviour for locations that were already hierarchical is unchanged: the same path is
resolved, the same
isDirectorycheck applies, and an equivalent absolutefile:URL isreturned.
Verifying this change
This change is a small bug fix in path resolution and is covered by the existing
flink-cdc-composertests. It can be reproduced without this patch by running any CDCpipeline in application mode with the connector JARs under
usrlib- the job fails duringcomposition with the stack trace above; with the patch the JAR path resolves and composition
proceeds.
Does this pull request potentially affect one of the following parts
@Public(Evolving): noapplication mode where it previously threw; no configuration change is required.
Documentation