Skip to content

Fix artifact staging windows filename - #39916

Open
Abacn wants to merge 2 commits into
apache:masterfrom
Abacn:fix-artifact-staging-windows-filename
Open

Fix artifact staging windows filename#39916
Abacn wants to merge 2 commits into
apache:masterfrom
Abacn:fix-artifact-staging-windows-filename

Conversation

@Abacn

@Abacn Abacn commented Aug 28, 2026

Copy link
Copy Markdown
Contributor

Please add a meaningful description for your change here

Fixes #39336

Port from #39363,


Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

  • Mention the appropriate issue in your description (for example: addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment fixes #<ISSUE NUMBER> instead.
  • Update CHANGES.md with noteworthy changes.
  • If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md

GitHub Actions Tests Status (on master branch)

Build python source distribution and wheels
Python tests
Java tests
Go tests

See CI.md for more information about GitHub Actions CI or the workflows README to see a list of phrases to trigger workflows.

@github-actions

Copy link
Copy Markdown
Contributor

Checks are failing. Will not request review until checks are succeeding. If you'd like to override that behavior, comment assign set of reviewers

@Abacn
Abacn force-pushed the fix-artifact-staging-windows-filename branch from 0ce85b9 to 62ee399 Compare August 29, 2026 01:13
@Abacn
Abacn force-pushed the fix-artifact-staging-windows-filename branch from 62ee399 to 3a7fd52 Compare August 29, 2026 02:19
@Abacn

Abacn commented Aug 29, 2026

Copy link
Copy Markdown
Contributor Author

R: @Eliaaazzz since you reviewed #39363

@github-actions

Copy link
Copy Markdown
Contributor

Stopping reviewer notifications for this pull request: review requested by someone other than the bot, ceding control. If you'd like to restart, comment assign set of reviewers

@codecov

codecov Bot commented Aug 29, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 25.00000% with 3 lines in your changes missing coverage. Please review.
✅ Project coverage is 58.34%. Comparing base (52d74b9) to head (3a7fd52).
⚠️ Report is 1 commits behind head on master.

Files with missing lines Patch % Lines
...s/fnexecution/artifact/ArtifactStagingService.java 33.33% 1 Missing and 1 partial ⚠️
...unners/fnexecution/environment/ProcessManager.java 0.00% 0 Missing and 1 partial ⚠️
Additional details and impacted files
@@             Coverage Diff              @@
##             master   #39916      +/-   ##
============================================
+ Coverage     55.06%   58.34%   +3.28%     
- Complexity     1722    13485   +11763     
============================================
  Files          1075     2568    +1493     
  Lines        171582   268734   +97152     
  Branches       1262    11029    +9767     
============================================
+ Hits          94474   156803   +62329     
- Misses        74838   105999   +31161     
- Partials       2270     5932    +3662     
Flag Coverage Δ
java 64.46% <25.00%> (-3.23%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@Eliaaazzz Eliaaazzz left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the ping, Yi, and thanks for picking this up together with the Windows CI wiring. I checked out the head commit on Windows 11 (JDK 11) and ran the module tests locally.

What I verified locally:

  • :runners:java-fn-execution:test passes at head, all 138 tests across 34 classes, including the new testStageArtifactsWithInvalidFilenameCharacters and the reworked ProcessManagerTest.
  • The new test genuinely catches a regression on Windows: if I revert the sanitization, it fails with ExecutionException: StatusRuntimeException: UNKNOWN out of offer, and the server side logs InvalidPathException: Illegal char <:>. With the windows-latest leg now running this module in java_tests.yml, that failure would show up in CI.
  • The environment sanitization resolves exactly the failure reported in #39336, and the env id in the new test matches the one from that report.

One substantive comment inline on createFilename, with a repro: the base component can still produce an invalid name for file-type artifacts.

On the two red checks:

  • beam_PreCommit_Flink_Container timed out in the Java Combine step. The last logged Gradle task is :runners:flink:2.0:jar and no test execution output appears before the timeout, while :runners:java-fn-execution:jar had completed cleanly a few minutes earlier. Recent master runs of that workflow are green, and I could not find a patch-related failure in the log.
  • codecov/patch flags three missing or partial lines, and they are the IS_OS_WINDOWS branches; the Java coverage upload runs on Ubuntu, which cannot take them.

Small question on java_tests.yml: the harness and runner steps used to carry if: always(), so they still ran when :sdks:java:core:test failed. Now an early failure skips them. Is that trade intentional, together with the single !success() upload?

List<String> components = Splitter.onPattern("[^A-Za-z-_.]]").splitToList(path);
String base = components.get(components.size() - 1);
if (IS_OS_WINDOWS) {
environment = WINDOWS_INVALID_CHARS.matcher(environment).replaceAll("_");

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The environment change addresses the reported case and preserves the Unicode behavior we discussed on #39363.

base on the line above can still break the filename on Windows, the same gap I noted on #39363: the splitter has an extra ] after the character class, so it only matches a disallowed character followed by a literal ]. For an ordinary absolute path it therefore does not split, and base stays the whole path (the stray ] is also written up in the last section of #39336). When a file-type artifact with a role other than staging_to carries an absolute client-side path, that path lands in the filename. One concrete in-tree producer is Python's _create_file_pip_requirements_artifact in stager.py, fed from PyPIArtifactRegistry through python_sdk_dependencies.

Verified at this head on Windows 11 by adding this to ArtifactStagingServiceTest:

@Test(timeout = 45_000)
public void testStageFileArtifactWithAbsolutePath() throws Exception {
  java.io.File source = tempFolder.newFile("real-artifact.bin");
  java.nio.file.Files.write(
      source.toPath(), "payload".getBytes(java.nio.charset.StandardCharsets.UTF_8));
  RunnerApi.ArtifactInformation fileArtifact =
      RunnerApi.ArtifactInformation.newBuilder()
          .setTypeUrn(ArtifactRetrievalService.FILE_ARTIFACT_URN)
          .setTypePayload(
              RunnerApi.ArtifactFilePayload.newBuilder()
                  .setPath(source.getAbsolutePath())
                  .build()
                  .toByteString())
          .setRoleUrn("beam:artifact:role:pip_requirements_file:v1")
          .build();
  stagingService.registerJob("fileToken", ImmutableMap.of("env", ImmutableList.of(fileArtifact)));
  ArtifactStagingService.offer(retrievalService, stagingStub, "fileToken");
  assertEquals(1, stagingService.getStagedArtifacts("fileToken").size());
}

It fails with ExecutionException: StatusRuntimeException: UNKNOWN out of offer, and the server side logs

InvalidPathException: Illegal char <:> at index 7: 1-env-C:\Users\...\real-artifact.bin

so base is the entire absolute path.

A one-line option: repair the splitter to "[^A-Za-z0-9-_.]". Java parses the dash after 0-9 as a literal there, and the pattern splits on both \ and /, so base drops the leading directories. With that change the repro above and the full :runners:java-fn-execution:test suite pass on my machine. The allowlist is ASCII-only, the tradeoff we discussed on #39363, though here it only trims base while the unique id prefix keeps names distinct. Also completely fine to keep this PR scoped to the environment id and handle base separately, whichever you prefer.

"spark.ui.enabled": "false",
"spark.ui.showConsoleProgress": "false",
// For Windows OS: see https://cwiki.apache.org/confluence/spaces/HADOOP2/pages/120730292/WindowsProblems
// 'hadoop.home.dir': "${rootDir.absolutePath}\\build\\hadoop-home",

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: is the commented-out property intentional? The line carries a tab indent, the only one in this file. If it is meant as a breadcrumb for running the Spark suites on Windows, the URL comment alone might be enough.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug]: Portable pipelines fail on Windows, artifact staging builds a filename containing a colon

3 participants