Skip to content

Commit 25e62c9

Browse files
dhergesdherges
authored
fixup! test(@angular/build): add library builder scaling benchmark suite
1 parent 90b8b5f commit 25e62c9

4 files changed

Lines changed: 565 additions & 565 deletions

File tree

‎scripts/benchmarks/library-builder/README.md‎

Lines changed: 119 additions & 119 deletions
Original file line numberDiff line numberDiff line change
@@ -1,119 +1,119 @@
1-
# Library Builds Scaling Benchmark
2-
3-
This benchmark measures build time metrics for Angular libraries.
4-
5-
## Context
6-
7-
Angular CLI is in the process of transitioning from `ng-packagr` to Angular CLI's native library builder.
8-
Both library build systems are registered Architect builders:
9-
- `@angular/build:ng-packagr`: runs library builds through the legacy `ng-packagr` build pipeline.
10-
- `@angular/build:library`: runs library builds through Angular CLI's native library builder.
11-
12-
This benchmark suite is designed to quantify performance impacts and ensure scalability.
13-
14-
### Previous Benchmarks
15-
16-
For context, when running a similar benchmark suite directly against `ng-packagr`, it hinted at a "U-curve" in build times,
17-
where build time per entry point in `ng-packagr` drops until entry point N (with N varying in the range 200..500).
18-
This shape indicates an initial overhead as a possible explanation for a small number of entry points.
19-
However, we discovered that `ng-packagr` build time per entry point increases
20-
as the number of entry points scale past a certain N (in the test runs N=200+).
21-
22-
[Landing Angular CLI's native `@angular/build:library` builder](https://github.com/angular/angular-cli/pull/34106#issuecomment-5744311681),
23-
indicated very favourable performance metrics for the new library builder that can be explained by number of
24-
architectural changes in the library build pipeline:
25-
- Shared Stylesheet Bundling
26-
- Public API Invalidation via Declaration Hashing
27-
- Dedicated Worker Pool for Production Builds
28-
- Single Pass Output Emission
29-
- Direct Async Pipeline
30-
31-
### Goals
32-
33-
This benchmark suite runs explorative research on the performance of library builds in Angular CLI.
34-
Interesting results from the previous benchmarks indicate 2 measurable goals:
35-
36-
- Measure the new library builder against itself: How does it scale with the number of entry points?
37-
- Enable A/B benchmarks of the `@angular/build:ng-packagr` builder against `@angular/build:library` builder: How much do library builds benefit from the transition to the native library builder?
38-
39-
As the previous benchmark runs hint at significant performance improvements in the new library builder,
40-
this benchmark suite aims to quantify the impact of that transition.
41-
42-
## Benchmark Design
43-
44-
This suite supports two primary layouts to test different filesystem and entry-point scaling scenarios:
45-
46-
- **Flat (`--layout=flat`)**: Generates $N$ secondary entry points as direct siblings, where $N$
47-
comes straight from `--count`.
48-
- Design: Each entry point is named `comp-i` (e.g., `comp-0001`, `comp-0002`).
49-
50-
- **Deep (`--layout=deep`)**: Generates $2^N$ secondary entry points in a binary tree structure.
51-
- Design: Creates a nested directory structure (`sub-a`/`sub-b` at each level).
52-
- Depth $N$ is not an independent sweep parameter -- see "Sweep rounds" below.
53-
54-
Each entry point consists of a `public-api.ts` and a component file with an inline template
55-
containing a bit of per-entry content derived from the entry's id (e.g. `<h2>Component no.
56-
0042</h2>`), so entries aren't byte-identical copies of each other.
57-
58-
### Sweep rounds
59-
60-
`--count` is the one sweep parameter: a comma-separated list of entry-point count targets. The
61-
default sweep spans small to large entry-point counts in one round, chosen to resolve the
62-
U-shaped scaling pattern observed with `ng-packagr` (see above):
63-
`10, 20, 50, 100, 200, 300, 400, 500, 1000, 2000`.
64-
65-
For `--layout=deep`, there's no "number of entry points" parameter directly -- a binary tree of
66-
depth $N$ always has exactly $2^N$ leaves/entry points. So `--count`'s targets are translated into
67-
tree depths: for each target `n`, the smallest `depth` with `2^depth >= n`, de-duplicated (e.g.
68-
`300`, `400`, and `500` all round up to the same `depth=9`, i.e. 512 entry points). This keeps
69-
both layouts comparable at roughly the same entry-point scale without exposing depth as a second,
70-
independently-tunable sweep axis. `--depth` still exists as an **explicit override** for
71-
`--layout=deep` -- pass it directly to sweep exact tree depths instead of entry-point targets --
72-
but it is not part of the default sweep design and is ignored for `--layout=flat`.
73-
74-
## Running the Benchmarks
75-
76-
```sh
77-
# Ensure the builder is built first
78-
pnpm build
79-
80-
# Run the benchmark
81-
pnpm admin benchmark library-builder [options]
82-
```
83-
84-
### Options
85-
86-
| Option | Description | Default |
87-
| :--------------------- | :----------------------------------------------- | :---------------------------------------- |
88-
| `--layout=<flat\|deep>` | Fixture layout | `flat` |
89-
| `--count=<n,n,...>` | Entry-point count targets, drives both layouts | `10,20,50,100,200,300,400,500,1000,2000` |
90-
| `--depth=<n,n,...>` | **`layout=deep` only.** Explicit tree depths, overriding the depths otherwise derived from `--count` | derived from `--count` |
91-
| `--iterations=<n>` | Measured iterations per size | `3` |
92-
| `--json` | Output raw measurements as JSON | N/A |
93-
94-
> `--count` and `--depth` are not independent sweep axes for `layout=deep`: `--count` is translated
95-
> into depths (smallest `depth` with `2^depth >= n` per target, de-duplicated); `--depth` only
96-
> matters when you want to bypass that translation and name exact tree depths yourself. Passing
97-
> both for `layout=deep` makes `--depth` win; `--depth` is ignored entirely for `layout=flat`.
98-
99-
### Example
100-
101-
Sweep the flat layout at a single size:
102-
103-
```sh
104-
pnpm admin benchmark library-builder --layout=flat --count=500
105-
```
106-
107-
## Benchmark Results
108-
109-
_(To be filled in once a sweep has been run and reviewed. Out of scope for this iteration.)_
110-
111-
## Future Work
112-
113-
- Style-flavor matrix: inline / inline-css / inline-scss / external / external-css / external-scss, swept across both layouts.
114-
- A/B comparison against `ng-packagr` (both are already registered Architect builders — `@angular/build:ng-packagr` and `@angular/build:library` — so this is a same-builder-package comparison, not a separate tool invocation).
115-
- Dependencies between entry points.
116-
- Shared stylesheet tokens imported by multiple components.
117-
- Watch mode.
118-
- Peak memory (`ru_maxrss`) and CPU time (`ru_utime` + `ru_stime`) metrics per build, in addition to wall-clock build time.
119-
- Library "taste" variants beyond plain flat/deep, e.g. flat/deep with inter-entry-point dependencies or shared SCSS tokens (currently out of scope, listed above).
1+
# Library Builds Scaling Benchmark
2+
3+
This benchmark measures build time metrics for Angular libraries.
4+
5+
## Context
6+
7+
Angular CLI is in the process of transitioning from `ng-packagr` to Angular CLI's native library builder.
8+
Both library build systems are registered Architect builders:
9+
- `@angular/build:ng-packagr`: runs library builds through the legacy `ng-packagr` build pipeline.
10+
- `@angular/build:library`: runs library builds through Angular CLI's native library builder.
11+
12+
This benchmark suite is designed to quantify performance impacts and ensure scalability.
13+
14+
### Previous Benchmarks
15+
16+
For context, when running a similar benchmark suite directly against `ng-packagr`, it hinted at a "U-curve" in build times,
17+
where build time per entry point in `ng-packagr` drops until entry point N (with N varying in the range 200..500).
18+
This shape indicates an initial overhead as a possible explanation for a small number of entry points.
19+
However, we discovered that `ng-packagr` build time per entry point increases
20+
as the number of entry points scale past a certain N (in the test runs N=200+).
21+
22+
[Landing Angular CLI's native `@angular/build:library` builder](https://github.com/angular/angular-cli/pull/34106#issuecomment-5744311681),
23+
indicated very favourable performance metrics for the new library builder that can be explained by number of
24+
architectural changes in the library build pipeline:
25+
- Shared Stylesheet Bundling
26+
- Public API Invalidation via Declaration Hashing
27+
- Dedicated Worker Pool for Production Builds
28+
- Single Pass Output Emission
29+
- Direct Async Pipeline
30+
31+
### Goals
32+
33+
This benchmark suite runs explorative research on the performance of library builds in Angular CLI.
34+
Interesting results from the previous benchmarks indicate 2 measurable goals:
35+
36+
- Measure the new library builder against itself: How does it scale with the number of entry points?
37+
- Enable A/B benchmarks of the `@angular/build:ng-packagr` builder against `@angular/build:library` builder: How much do library builds benefit from the transition to the native library builder?
38+
39+
As the previous benchmark runs hint at significant performance improvements in the new library builder,
40+
this benchmark suite aims to quantify the impact of that transition.
41+
42+
## Benchmark Design
43+
44+
This suite supports two primary layouts to test different filesystem and entry-point scaling scenarios:
45+
46+
- **Flat (`--layout=flat`)**: Generates $N$ secondary entry points as direct siblings, where $N$
47+
comes straight from `--count`.
48+
- Design: Each entry point is named `comp-i` (e.g., `comp-0001`, `comp-0002`).
49+
50+
- **Deep (`--layout=deep`)**: Generates $2^N$ secondary entry points in a binary tree structure.
51+
- Design: Creates a nested directory structure (`sub-a`/`sub-b` at each level).
52+
- Depth $N$ is not an independent sweep parameter -- see "Sweep rounds" below.
53+
54+
Each entry point consists of a `public-api.ts` and a component file with an inline template
55+
containing a bit of per-entry content derived from the entry's id (e.g. `<h2>Component no.
56+
0042</h2>`), so entries aren't byte-identical copies of each other.
57+
58+
### Sweep rounds
59+
60+
`--count` is the one sweep parameter: a comma-separated list of entry-point count targets. The
61+
default sweep spans small to large entry-point counts in one round, chosen to resolve the
62+
U-shaped scaling pattern observed with `ng-packagr` (see above):
63+
`10, 20, 50, 100, 200, 300, 400, 500, 1000, 2000`.
64+
65+
For `--layout=deep`, there's no "number of entry points" parameter directly -- a binary tree of
66+
depth $N$ always has exactly $2^N$ leaves/entry points. So `--count`'s targets are translated into
67+
tree depths: for each target `n`, the smallest `depth` with `2^depth >= n`, de-duplicated (e.g.
68+
`300`, `400`, and `500` all round up to the same `depth=9`, i.e. 512 entry points). This keeps
69+
both layouts comparable at roughly the same entry-point scale without exposing depth as a second,
70+
independently-tunable sweep axis. `--depth` still exists as an **explicit override** for
71+
`--layout=deep` -- pass it directly to sweep exact tree depths instead of entry-point targets --
72+
but it is not part of the default sweep design and is ignored for `--layout=flat`.
73+
74+
## Running the Benchmarks
75+
76+
```sh
77+
# Ensure the builder is built first
78+
pnpm build
79+
80+
# Run the benchmark
81+
pnpm admin benchmark library-builder [options]
82+
```
83+
84+
### Options
85+
86+
| Option | Description | Default |
87+
| :--------------------- | :----------------------------------------------- | :---------------------------------------- |
88+
| `--layout=<flat\|deep>` | Fixture layout | `flat` |
89+
| `--count=<n,n,...>` | Entry-point count targets, drives both layouts | `10,20,50,100,200,300,400,500,1000,2000` |
90+
| `--depth=<n,n,...>` | **`layout=deep` only.** Explicit tree depths, overriding the depths otherwise derived from `--count` | derived from `--count` |
91+
| `--iterations=<n>` | Measured iterations per size | `3` |
92+
| `--json` | Output raw measurements as JSON | N/A |
93+
94+
> `--count` and `--depth` are not independent sweep axes for `layout=deep`: `--count` is translated
95+
> into depths (smallest `depth` with `2^depth >= n` per target, de-duplicated); `--depth` only
96+
> matters when you want to bypass that translation and name exact tree depths yourself. Passing
97+
> both for `layout=deep` makes `--depth` win; `--depth` is ignored entirely for `layout=flat`.
98+
99+
### Example
100+
101+
Sweep the flat layout at a single size:
102+
103+
```sh
104+
pnpm admin benchmark library-builder --layout=flat --count=500
105+
```
106+
107+
## Benchmark Results
108+
109+
_(To be filled in once a sweep has been run and reviewed. Out of scope for this iteration.)_
110+
111+
## Future Work
112+
113+
- Style-flavor matrix: inline / inline-css / inline-scss / external / external-css / external-scss, swept across both layouts.
114+
- A/B comparison against `ng-packagr` (both are already registered Architect builders — `@angular/build:ng-packagr` and `@angular/build:library` — so this is a same-builder-package comparison, not a separate tool invocation).
115+
- Dependencies between entry points.
116+
- Shared stylesheet tokens imported by multiple components.
117+
- Watch mode.
118+
- Peak memory (`ru_maxrss`) and CPU time (`ru_utime` + `ru_stime`) metrics per build, in addition to wall-clock build time.
119+
- Library "taste" variants beyond plain flat/deep, e.g. flat/deep with inter-entry-point dependencies or shared SCSS tokens (currently out of scope, listed above).

0 commit comments

Comments
 (0)