Fix artifact staging windows filename - #39916
Conversation
|
Checks are failing. Will not request review until checks are succeeding. If you'd like to override that behavior, comment |
0ce85b9 to
62ee399
Compare
62ee399 to
3a7fd52
Compare
|
R: @Eliaaazzz since you reviewed #39363 |
|
Stopping reviewer notifications for this pull request: review requested by someone other than the bot, ceding control. If you'd like to restart, comment |
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## master #39916 +/- ##
============================================
+ Coverage 55.06% 58.34% +3.28%
- Complexity 1722 13485 +11763
============================================
Files 1075 2568 +1493
Lines 171582 268734 +97152
Branches 1262 11029 +9767
============================================
+ Hits 94474 156803 +62329
- Misses 74838 105999 +31161
- Partials 2270 5932 +3662
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Eliaaazzz
left a comment
There was a problem hiding this comment.
Thanks for the ping, Yi, and thanks for picking this up together with the Windows CI wiring. I checked out the head commit on Windows 11 (JDK 11) and ran the module tests locally.
What I verified locally:
:runners:java-fn-execution:testpasses at head, all 138 tests across 34 classes, including the newtestStageArtifactsWithInvalidFilenameCharactersand the reworkedProcessManagerTest.- The new test genuinely catches a regression on Windows: if I revert the sanitization, it fails with
ExecutionException: StatusRuntimeException: UNKNOWNout ofoffer, and the server side logsInvalidPathException: Illegal char <:>. With the windows-latest leg now running this module injava_tests.yml, that failure would show up in CI. - The environment sanitization resolves exactly the failure reported in #39336, and the env id in the new test matches the one from that report.
One substantive comment inline on createFilename, with a repro: the base component can still produce an invalid name for file-type artifacts.
On the two red checks:
beam_PreCommit_Flink_Containertimed out in the Java Combine step. The last logged Gradle task is:runners:flink:2.0:jarand no test execution output appears before the timeout, while:runners:java-fn-execution:jarhad completed cleanly a few minutes earlier. Recent master runs of that workflow are green, and I could not find a patch-related failure in the log.codecov/patchflags three missing or partial lines, and they are theIS_OS_WINDOWSbranches; the Java coverage upload runs on Ubuntu, which cannot take them.
Small question on java_tests.yml: the harness and runner steps used to carry if: always(), so they still ran when :sdks:java:core:test failed. Now an early failure skips them. Is that trade intentional, together with the single !success() upload?
| List<String> components = Splitter.onPattern("[^A-Za-z-_.]]").splitToList(path); | ||
| String base = components.get(components.size() - 1); | ||
| if (IS_OS_WINDOWS) { | ||
| environment = WINDOWS_INVALID_CHARS.matcher(environment).replaceAll("_"); |
There was a problem hiding this comment.
The environment change addresses the reported case and preserves the Unicode behavior we discussed on #39363.
base on the line above can still break the filename on Windows, the same gap I noted on #39363: the splitter has an extra ] after the character class, so it only matches a disallowed character followed by a literal ]. For an ordinary absolute path it therefore does not split, and base stays the whole path (the stray ] is also written up in the last section of #39336). When a file-type artifact with a role other than staging_to carries an absolute client-side path, that path lands in the filename. One concrete in-tree producer is Python's _create_file_pip_requirements_artifact in stager.py, fed from PyPIArtifactRegistry through python_sdk_dependencies.
Verified at this head on Windows 11 by adding this to ArtifactStagingServiceTest:
@Test(timeout = 45_000)
public void testStageFileArtifactWithAbsolutePath() throws Exception {
java.io.File source = tempFolder.newFile("real-artifact.bin");
java.nio.file.Files.write(
source.toPath(), "payload".getBytes(java.nio.charset.StandardCharsets.UTF_8));
RunnerApi.ArtifactInformation fileArtifact =
RunnerApi.ArtifactInformation.newBuilder()
.setTypeUrn(ArtifactRetrievalService.FILE_ARTIFACT_URN)
.setTypePayload(
RunnerApi.ArtifactFilePayload.newBuilder()
.setPath(source.getAbsolutePath())
.build()
.toByteString())
.setRoleUrn("beam:artifact:role:pip_requirements_file:v1")
.build();
stagingService.registerJob("fileToken", ImmutableMap.of("env", ImmutableList.of(fileArtifact)));
ArtifactStagingService.offer(retrievalService, stagingStub, "fileToken");
assertEquals(1, stagingService.getStagedArtifacts("fileToken").size());
}It fails with ExecutionException: StatusRuntimeException: UNKNOWN out of offer, and the server side logs
InvalidPathException: Illegal char <:> at index 7: 1-env-C:\Users\...\real-artifact.bin
so base is the entire absolute path.
A one-line option: repair the splitter to "[^A-Za-z0-9-_.]". Java parses the dash after 0-9 as a literal there, and the pattern splits on both \ and /, so base drops the leading directories. With that change the repro above and the full :runners:java-fn-execution:test suite pass on my machine. The allowlist is ASCII-only, the tradeoff we discussed on #39363, though here it only trims base while the unique id prefix keeps names distinct. Also completely fine to keep this PR scoped to the environment id and handle base separately, whichever you prefer.
| "spark.ui.enabled": "false", | ||
| "spark.ui.showConsoleProgress": "false", | ||
| // For Windows OS: see https://cwiki.apache.org/confluence/spaces/HADOOP2/pages/120730292/WindowsProblems | ||
| // 'hadoop.home.dir': "${rootDir.absolutePath}\\build\\hadoop-home", |
There was a problem hiding this comment.
nit: is the commented-out property intentional? The line carries a tab indent, the only one in this file. If it is meant as a breadcrumb for running the Spark suites on Windows, the URL comment alone might be enough.
Please add a meaningful description for your change here
Fixes #39336
Port from #39363,
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, commentfixes #<ISSUE NUMBER>instead.CHANGES.mdwith noteworthy changes.See the Contributor Guide for more tips on how to make review process smoother.
To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md
GitHub Actions Tests Status (on master branch)
See CI.md for more information about GitHub Actions CI or the workflows README to see a list of phrases to trigger workflows.