Repository navigation
Introduce secondary report format to enable caching of plugin results #1048
Description
User story
As a developer I want to be able to cache as much as possible with Nx and leverage affected
Why Commit Info Breaks Caching
To get reliable caching in tools like Nx, task inputs must stay stable across commits — unless the output truly depends on them.
Example ESLint:
ESLint can be cached effectively when its inputs are:
- The ESLint version
- Source files of the project
- Dependency .d.ts files
In this setup, even if you're on a different commit or branch, ESLint won’t re-run unless something relevant changes. This allows maximum cache reuse.
Note
Using Commit SHA as Input
If you include the commit SHA in:
- Task inputs
- Generated files
- Plugin outputs
Then every commit creates a unique input hash, and ESLint (or any task) loses its cache — even when the code hasn’t changed.
Suggestion
Legend:
- 🐳 Owned by Nx — Nx or other external caching solutions
- 🔌 Owned by plugin creator — depends on code changes
- ☑️ Owned by CodePushup — depends on code changes
- 🔧 Code-related — depends on code changes
- ⛓️ Commit-related — depends on commit (includes code changes)
- ⚡ Fast — fast enough to optionally skip caching
- 💾 Cached — slow or expensive, should be cached
- 🚫 No Cache — intentionally avoids caching
New CLI options for command collect
| Option | Type | Default | Description |
|---|---|---|---|
--persist.report |
boolean |
true |
Will not produce report.json. Normally in combination with --cache |
--cache.write |
boolean |
false |
If true it will generate a {persist.outputDir}/<plugin-slug>/audit-outputs.json file containing the audit output data of a plugin |
--cache.read |
boolean |
false |
If true it will try to read the plugins audit outputs for file system. If not available it will run the plugin to get the audit outputs on the fly. If |
Collect Flow - Generation Phase
Command: npx @code-pushup/cli collect --cache.write --no-report --onlyPlugins eslint
-
For each plugin listed in
--onlyPlugins:IF
--cache.writeis given it will write the audit outputs data to{persist.outputDir}/<plugin-slug>/audit-outputs.json(💾☑️).
This is done in executePlugin after the execution. -
After the plugins are run
IF --no-report is set, no final report persist.outputDir/report.json is created. (☑️)
Collect Flow - Aggregation Phase
Command: npx @code-pushup/cli collect --cache.read
IF --cache.read is given the CLI checks for existing plugin audit outputs under persist.outputDir/<plugin-slug>/audit-outputs.json and if given it will take the data instead of executing the plugin runner.
This is done in executePlugin before the execution.
ELSE:
IF generateArtifactsCommand is defined
→ Run the command e.g., npx nx run-many ... to produce the raw artifacts (🐳)
Optional Generate plugin artifacts (💾 🔧 🔌)
Generate the plugin report and persist it to persist.outputDir/<plugin-slug>/audit-outputs.json (💾 🔧 ☑️)
In executePlugin we save the plugin report as AuditOutputs.
- After all plugins are processed, the CLI will combine reports into the final report
persist.outputDir/report.json(⛓️ ⚡ 🚫 🔧 ☑️)
Code Pushup Config
{
plugins: [
await coveragePlugin({
generateArtifactsCommand: 'npx nx run-many --project lib-a --targets unit-test int-test',
reports: ['packages/lib-a/coverage/unit-test/lcov.info', 'packages/lib-a/coverage/int-test/lcov.info'],
}),
await nxPerfPlugin({
audits: ['project-graph-time'],
}),
],
}Targets
{
"name": "lib-a",
"targets": {
"int-test": {
"cache": true,
"outputs": ["{options.coverage.reportsDirectory}"],
"executor": "@nx/vite:test",
"options": {
"configFile": "packages/lib-a/vitest.int.config.ts",
"coverage.reportsDirectory": "{projectRoot}/coverage/int-test"
}
},
"unit-test": {
"cache": true,
"outputs": ["{options.coverage.reportsDirectory}"],
"executor": "@nx/vite:test",
"options": {
"configFile": "packages/lib-a/vitest.unit.config.ts",
"coverage.reportsDirectory": "{projectRoot}/coverage/unit-test"
}
},
"code-pushup:coverage": {
"dependsOn": ["unit-test", "int-test"],
"cache": true,
"inputs": ["{coverage-speciftic-globs}"],
"outputs": ["{options.outputPath}"],
"executor": "@code-pushup/nx-plugin:cli",
"options": {
"report": false,
"cache.write": true,
"onlyPlugins": "coverage",
"persist.outputDir": "{projectRoot}/.code-pushup/coverage",
"outputPath": "{projectRoot}/.code-pushup/coverage/<plugin-slug>/plugin-report.json",
}
},
"//":"This target could be completely skipped as we d not cache anything here",
"code-pushup:nx-perf": {
"cache": false,
"executor": "@code-pushup/nx-plugin:cli",
"options": {
"report": false,
"onlyPlugins": "nx-perf"
"persist.outputDir": "{projectRoot}/.code-pushup/nx-perf"
}
},
"code-pushup": {
"dependsOn": ["code-pushup:coverage", "code-pushup:nx-perf"],
"cache": false,
"outputs": ["{options.outputPath}"],
"executor": "@code-pushup/nx-plugin:cli",
"options": {
"cache.read": true,
"persist.outputDir": "{projectRoot}/.code-pushup",
"outputPath": "{projectRoot}/.code-pushup/report.*",
}
}
}
}Nx Task Graph
graph TD
A[lib-a:code-pushup ⚡ ⛓️ 🚫] --> B[lib-a:code-pushup:coverage 💾 🔧 ☑️]
A --> E[lib-a:code-pushup:nx-perf 🚫 🔌]
B --> C[lib-a:unit-test 💾 🔧 🐳]
B --> D[lib-a:int-test 💾 🔧 🐳]
Acceptance criteria
- New CLI options
--report- feat: add report option to cli #1058- New report format
{persist.outputDir}/<plugin-slug>/runner-output.json
- New report format
- New CLI options
--cache.writeand--cache.readfeat: add audit output caching for execute plugin #1057- Collect looks for plugin reports first and only runs if they are missing
- CLI respects report and caching options feat: add caching options to cli #1059
Implementation details
No response
Activity
beaussan commentedon Aug 5, 2025
The only thing I would say would be to introduce a dedicated persist option, instead of reusing
persist.outputDir. Otherwise code-pushup can't know where plugin cached outputs (audits, issues, etc) are locatedBioPhoton commentedon Aug 5, 2025
We could make it owned by the CLI and use one of the patterns:
<persist.outputDir>/<plugin-slug>-plugin-report.json<persist.outputDir>/<plugin-slug>/plugin-report.json<persist.outputDir>/plugin-report.<plugin-slug>.jsonAlso the output would be this structure: https://github.com/code-pushup/cli/blob/main/packages/models/docs/models-reference.md#pluginreport
beaussan commentedon Aug 6, 2025
I think this step should only be run when running explicitly a plugin directly, otherwise this could lead to weird state where things are not re ran properly. A bit like the new ci setup,
Default is safe, but slow. If you want fast, you have to deal with caching per plugin if you need so, but beware that if you don't cleenup properly, or not cache properly, your data will be stalematejchalk commentedon Aug 7, 2025
To cache plugins, we can only cache the runner output, not the whole
PluginReport. All the plugin metadata is loaded when thecode-pushup.config.tsis imported, only the runner is executed by the CLI, theAuditOutput[]is then combined with all the metadata to create thePluginReport. You need to load the config file before you even know which plugins exist, so how could you cache the metadata? I don't think you need to anyway, the slow part is always in the runner.BioPhoton commentedon Aug 7, 2025
Changed type. ☝️
BioPhoton commentedon Aug 7, 2025
Added CLI options
[-]Introduce secondary report format to enable caching of plugin resutlts[/-][+]Introduce secondary report format to enable caching of plugin results[/+]feat: add audit output caching for execute plugin (#1057)
feat: add caching options to cli (#1059)