Skip to content

Implement native clang launcher#27401

Open
sbc100 wants to merge 2 commits into
emscripten-core:mainfrom
sbc100:native_clang_impl
Open

Implement native clang launcher#27401
sbc100 wants to merge 2 commits into
emscripten-core:mainfrom
sbc100:native_clang_impl

Conversation

@sbc100

@sbc100 sbc100 commented Jul 22, 2026

Copy link
Copy Markdown
Collaborator

Implement Phase 1 of the native launcher design document. Provides
high-performance native C++ compiler drivers (emcc, em++) using C++20
that directly invoke Clang for compile steps (-c, -S, -E) and seamlessly
fall back to python (emcc.py / em++.py) for link steps or complex
post-processing options.

Output executables are placed in ./bin using CMake and Ninja. Adds CI
matrix testing for Linux, macOS, and Windows.

These new binaries are currently 100% optional and the python versions
can still be used without building them.

Tested using embuilder build libc --force (compiling 1,075
files sequentially with EMCC_CORES=1, EMCC_USE_NINJA=0, and
EMCC_BATCH_BUILD=0). Note that this is likely an outlier since
its perfectly suited to demonstrating to speedup from this change (i.e.
lots of small source files and no linking).

Platform Before (Python Baseline) After (Native Launcher) Improvement Speedup
Linux 181.96 s (169.3 ms/file) 64.82 s (60.3 ms/file) -117.14 s (-109.0 ms/file) 2.81x
Windows 343.90 s (319.9 ms/file) 105.12 s (97.8 ms/file) -238.78 s (-222.1 ms/file) 3.27x
macOS 162.08 s (150.6 ms/file) 64.73 s (60.1 ms/file) -97.35 s (-90.5 ms/file) 2.50x

See: #26453

@sbc100
sbc100 marked this pull request as draft July 22, 2026 00:32
@sbc100 sbc100 changed the title [native] Implement native clang launcher frontend Implement native clang launcher frontend Jul 22, 2026
@sbc100 sbc100 changed the title Implement native clang launcher frontend Implement native clang launcher Jul 22, 2026
@sbc100
sbc100 force-pushed the native_clang_impl branch 6 times, most recently from 198f9c8 to c5fd0d6 Compare July 22, 2026 21:53
@sbc100
sbc100 marked this pull request as ready for review July 22, 2026 21:54
@sbc100
sbc100 force-pushed the native_clang_impl branch from c5fd0d6 to 3d058e6 Compare July 22, 2026 22:04
@sbc100
sbc100 requested a review from dschuff July 22, 2026 22:05
@sbc100
sbc100 force-pushed the native_clang_impl branch 4 times, most recently from 470205f to bac6b35 Compare July 22, 2026 23:22
@sbc100
sbc100 requested a review from brendandahl July 22, 2026 23:32
@dschuff

dschuff commented Jul 22, 2026

Copy link
Copy Markdown
Member

Here are some high level comments:

  1. Do we need response file support? Super long command lines are more common at link time, but the windows cmdline length limit is pretty low, so maybe a long path with a few include directives would be enough to push us over even at compile time.
  2. If we are using C++20, we could use std::string_view instead of std::string in a lot of places to avoid a lot of allocations and copies in string operations, e.g. is_emscripten_only_warning, is_dash_s_setting, and other places. I got a bunch of other similar suggestions from Gemini for how to reduce allocations and redundant filesystem operations that might be worth considering to reduce overhead on startup.

Comment thread tools/emcc_native/main.cpp Outdated
Comment thread tools/emcc_native/main.cpp Outdated
Comment thread tools/emcc_native/driver.cpp Outdated
Comment thread tools/emcc_native/driver.cpp
@sbc100
sbc100 force-pushed the native_clang_impl branch 4 times, most recently from 06b94ef to 9fec6c9 Compare July 23, 2026 00:22
@sbc100
sbc100 requested a review from dschuff July 23, 2026 00:23
@sbc100
sbc100 force-pushed the native_clang_impl branch from 9fec6c9 to b5f76dc Compare July 23, 2026 00:36
Comment thread tools/emcc_native/driver.cpp
Comment thread tools/emcc_native/driver.cpp
Comment thread tools/emcc_native/driver.cpp Outdated
@sbc100
sbc100 force-pushed the native_clang_impl branch 4 times, most recently from 9ac1ab1 to c94e316 Compare July 23, 2026 19:16
@sbc100
sbc100 force-pushed the native_clang_impl branch 7 times, most recently from 0abdc68 to b7f2e37 Compare July 23, 2026 22:37
@sbc100

sbc100 commented Jul 23, 2026

Copy link
Copy Markdown
Collaborator Author

Benchmark results for are in.. libc builds are much faster on all platforms! Yay

@sbc100
sbc100 requested review from dschuff, juj and kripken July 23, 2026 22:39
@sbc100

sbc100 commented Jul 23, 2026

Copy link
Copy Markdown
Collaborator Author

I think once we land this and we start using it everywhere we can just remove the old shell/bat/ps1/pylauncher launchers.

@sbc100
sbc100 force-pushed the native_clang_impl branch 2 times, most recently from 13461be to 9e62051 Compare July 24, 2026 01:04
Implement Phase 1 of the native launcher design document. Provides
high-performance native C++ compiler drivers (emcc, em++) using C++20
that directly invoke Clang for compile steps (-c, -S, -E) and seamlessly
fall back to python (emcc.py / em++.py) for link steps or complex
post-processing options.

Output executables are placed in ./bin using CMake and Ninja. Adds CI
matrix testing for Linux, macOS, and Windows.

These new binaries are currently 100% optional and the python versions
can still be used without building them.

Tested using `embuilder build libc --force` on Linux CI (compiling 1,075 files sequentially with `EMCC_CORES=1`, `EMCC_USE_NINJA=0`, and `EMCC_BATCH_BUILD=0`):

| Metric | Before (Python Baseline) | After (Native Launcher) | Improvement |
| :--- | :---: | :---: | :---: |
| **Total Time** | **181.96 s** (3m 2s) | **64.82 s** (1m 5s) | **-117.14 s (64.4% faster)** |
| **Time per File** | 169.3 ms | 60.3 ms | **-109.0 ms / invocation** |
| **Speedup Factor** | 1.0x | **2.81x** | — |

**Key Takeaways:**

* **64.4% Reduction in Build Time:** Bypassing Python interpreter startup
  shaves off **~109.0 ms of overhead per compiler invocation** on Linux CI
  (a **2.81x speedup**).
* **Windows Impact:** Because process spawning and `python.exe` startup
  carry significantly higher overhead on Windows than on Linux, the
  percentage speedup on Windows CI is expected to be even larger.

See: emscripten-core#26453
@sbc100
sbc100 force-pushed the native_clang_impl branch from 9e62051 to 3a2cafc Compare July 24, 2026 01:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants