Develocity integration by ribafish · Pull Request #9167 · apache/cloudstack

ribafish · 2024-06-03T21:31:53Z

@GutoVeronezi, it was nice meeting you at Community over Code today. This PR will enable you to publish Build Scans to develocity.apache.org as discussed.

Description

This PR publishes a build scan for every (authenticated) CI build and for every local build from an authenticated Apache committer. The build will not fail if publishing fails. Local and remote caching was left disabled on this PR by design so that the build is not affected by this change.

The build scans of the Apache Cloudstack project are published to the Develocity instance at develocity.apache.org, hosted by the Apache Software Foundation and run in partnership between the ASF and Gradle. This Develocity instance has all features and extensions enabled and is freely available for use by the Apache Cloudstack project and all other Apache projects.

On this Develocity instance, Apache Cloudstack will have access not only to all of the published build scans but other aggregate data features such as:

Dashboards to view all historical build scans, along with performance trends over time
Build failure analytics for enhanced investigation and diagnosis of build failures
Test failure analytics to better understand trends and causes around slow, failing, and flaky tests

Please let me know if there are any questions about the value of Develocity or the changes in this pull request and I’d be happy to address them.

Types of changes

Breaking change (fix or feature that would cause existing functionality to change)
New feature (non-breaking change which adds functionality)
Bug fix (non-breaking change which fixes an issue)
Enhancement (improves an existing feature and functionality)
Cleanup (Code refactoring and cleanup, that may add test cases)
build/CI

boring-cyborg · 2024-06-03T21:31:58Z

Congratulations on your first Pull Request and welcome to the Apache CloudStack community! If you have any issues or are unsure about any anything please check our Contribution Guide (https://github.com/apache/cloudstack/blob/main/CONTRIBUTING.md)
Here are some useful points:

In case of a new feature add useful documentation (raise doc PR at https://github.com/apache/cloudstack-documentation)
Be patient and persistent. It might take some time to get a review or get the final approval from the committers.
Pay attention to the quality of your code, ensure tests are passing and your PR doesn't have conflicts.
Please follow ASF Code of Conduct for all communication including (but not limited to) comments on Pull Requests, Issues, Mailing list and Slack.
Be sure to read the CloudStack Coding Conventions.
Apache CloudStack is a community-driven project and together we are making it better 🚀.
In case of doubts contact the developers at:
Mailing List: dev@cloudstack.apache.org (https://cloudstack.apache.org/mailing-lists.html)
Slack: https://apachecloudstack.slack.com/

codecov · 2024-06-04T06:12:26Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 20.21%. Comparing base (68cff97) to head (fc7b58a).

Additional details and impacted files

@@            Coverage Diff            @@
##               main    #9167   +/-   ##
=========================================
  Coverage     20.21%   20.21%           
+ Complexity    20750    20747    -3     
=========================================
  Files          6426     6426           
  Lines        580181   580181           
  Branches      71033    71033           
=========================================
+ Hits         117287   117289    +2     
+ Misses       450180   450178    -2     
  Partials      12714    12714

Flag	Coverage Δ
uitests	`3.69% <ø> (ø)`
unittests	`21.51% <ø> (+<0.01%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

GutoVeronezi · 2024-06-05T10:55:43Z

@ribafish, thanks for the PR; it was nice to meet you at the conference. Now we will run the tests and, as soon we meet the requirements, we will merge the PR.

GutoVeronezi · 2024-06-05T10:55:53Z

@blueorangutan package

blueorangutan · 2024-06-05T10:58:04Z

@GutoVeronezi a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2024-06-05T12:18:56Z

Packaging result [SF]: ✔️ el7 ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 9793

DaanHoogland · 2024-06-06T07:21:45Z

@GutoVeronezi what is your intention with this? Are you proposing to base some policy on this?

Also,
do we need more testing than the github actions and build/packaging?

GutoVeronezi · 2024-06-07T06:51:21Z

@GutoVeronezi what is your intention with this? Are you proposing to base some policy on this?

The main goal is to identify slow and flaky tests, so we can improve our code and workflow. Having those metrics can also help us to identify other problems that might be there and we are not seeing.

Also, do we need more testing than the github actions and build/packaging?

Not sure if more, but well structured.

DaanHoogland · 2024-06-07T08:23:16Z

Also, do we need more testing than the github actions and build/packaging?

Not sure if more, but well structured.

@GutoVeronezi , my communication skill are way below par. I meant to ask if we need more testing on this PR. You are absolutely right in your reply and it is valid with repect to the question I posed, just not to the one I intended ;)

GutoVeronezi · 2024-06-07T15:37:01Z

Also, do we need more testing than the github actions and build/packaging?

Not sure if more, but well structured.

@GutoVeronezi , my communication skill are way below par. I meant to ask if we need more testing on this PR. You are absolutely right in your reply and it is valid with repect to the question I posed, just not to the one I intended ;)

I see hehe

It should not affect the system itself; however, I would run it just for sanity.

vishesh92 · 2024-06-08T08:38:52Z

@ribafish Thanks for PR.

I don't remember exactly, but there was some issue related to security in github actions resulting in leaking of secrets. One of the reasons we disabled sonar on PRs from forks.
@DaanHoogland Do you think that might be an issue here as well?

weizhouapache · 2024-06-08T10:39:50Z

@ribafish Thanks for PR.

I don't remember exactly, but there was some issue related to security in github actions resulting in leaking of secrets. One of the reasons we disabled sonar on PRs from forks.
@DaanHoogland Do you think that might be an issue here as well?

It was because of target in the github actions. I think this pr doesn't have the issue.
I have no idea what this pr is used for. We do not use gradle at all. Let's see

ribafish · 2024-06-10T08:45:57Z

Hi @vishesh92 and @weizhouapache

This PR enables you to push build scans to Apache Develocity at ge.apache.org. These capture metadata about your build, which can then be used to monitor and improve the build performance, test times and fix flaky tests. It is also a prerequisite for other build performance improvements, such as using local and remote build cache, test distribution or predictive test selection (not part of this PR).

Develocity supports multiple build tools; there are also several Apache projects built with Maven that use it already, such as Camel, Pulsar, Creadur RAT, etc. -> you can see their build scans at ge.apache.org with the Maven filter applied.

The information captured includes the execution plan and timings, tests executed, dependencies and their versions, etc., but it does not include information such as the full list of env vars or the full cli command used to start a build unless added explicitly via custom values.

weizhouapache · 2024-06-10T09:40:20Z

Hi @vishesh92 and @weizhouapache

This PR enables you to push build scans to Apache Develocity at ge.apache.org. These capture metadata about your build, which can then be used to monitor and improve the build performance, test times and fix flaky tests. It is also a prerequisite for other build performance improvements, such as using local and remote build cache, test distribution or predictive test selection (not part of this PR).

Develocity supports multiple build tools; there are also several Apache projects built with Maven that use it already, such as Camel, Pulsar, Creadur RAT, etc. -> you can see their build scans at ge.apache.org with the Maven filter applied.

The information captured includes the execution plan and timings, tests executed, dependencies and their versions, etc., but it does not include information such as the full list of env vars or the full cli command used to start a build unless added explicitly via custom values.

@ribafish
thanks for the information
as I understand, with this pr, the info of github actions will be uploaded to ge.apache.org for analysis, right ?

do we need to ask ASF infra to add the secret GE_ACCESS_TOKEN ?

ribafish · 2024-06-10T09:48:02Z

It is set up that any authenticated build will get published - at the moment, only CI jobs are authenticated and ASF infra already added GE_ACCESS_TOKEN to the organization secrets, so there's no need to do anything else. If you wanted to also publish local build scans, all you need to do is to authenticate with ge.apache.org, which you can do like this. To sign in to ge.apache.org you can use your ASF credentials, as your LDAP is set up with Develocity.

weizhouapache · 2024-06-10T09:58:24Z

It is set up that any authenticated build will get published - at the moment, only CI jobs are authenticated and ASF infra already added GE_ACCESS_TOKEN to the organization secrets, so there's no need to do anything else. If you wanted to also publish local build scans, all you need to do is to authenticate with ge.apache.org, which you can do like this. To sign in to ge.apache.org you can use your ASF credentials, as your LDAP is set up with Develocity.

cool, thanks @ribafish

I've got an access key via command mvn com.gradle:develocity-maven-extension:provision-access-key

weizhouapache

code lgtm

weizhouapache · 2024-06-10T18:32:20Z

@ribafish
quick question, is it possible to remove some build scans ?

clayburn · 2024-06-10T18:34:36Z

@weizhouapache - an admin can do this. Are you referring to the 4 local build scans from this project? If so, I can ensure this happens

weizhouapache · 2024-06-10T19:02:58Z

@weizhouapache - an admin can do this. Are you referring to the 4 local build scans from this project? If so, I can ensure this happens

@clayburn
Yes, can you please remove them?
thanks

clayburn · 2024-06-10T19:27:19Z

Done

DaanHoogland · 2024-06-11T19:15:40Z

@blueorangutan package

blueorangutan · 2024-06-11T19:18:04Z

@DaanHoogland a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

DaanHoogland · 2026-08-16T17:22:11Z

@ribafish , sorry to start looking at the changes without context. I like the premise and would see this merged. I think I ran into a github glitch (in the matrix) meaning you can ignore my prior two comments.

DaanHoogland

lgtm

Copilot

Warning

Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.

Copilot review overview

3 open findings

The #{{ ... }} syntax looks malformed for Develocity's expression interpolation (typically `#{… · New
Setting DEVELOCITY_ACCESS_KEY at the workflow level makes the secret available to every step in… · New
The same env: DEVELOCITY_ACCESS_KEY block is duplicated across several workflows in this PR. To… · New

1 resolved since last review

<buildCache><remote> is explicitly disabled (<enabled>false</enabled>), so <storeEnabled>…

🧠 Review effort: Lite

Copilot

Warning

Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.

Copilot review overview

7 open findings

The value expression syntax on <ipAddresses> appears invalid (#{{ ... }}). Develocity Maven… · New
The #{{ ... }} syntax looks malformed for Develocity's expression interpolation (typically `#{…
Setting DEVELOCITY_ACCESS_KEY at workflow scope makes it available to all jobs/steps (including… · New
The PR description says a build scan is published for every CI build, but onlyIf set to… · New
With <remote><enabled>false</enabled>, the <storeEnabled> setting is ineffective and may be… · New
Setting DEVELOCITY_ACCESS_KEY at the workflow level makes the secret available to every step in…
The same env: DEVELOCITY_ACCESS_KEY block is duplicated across several workflows in this PR. To…

🧠 Review effort: Lite

Signed-off-by: Gasper Kojek <gkojek@gradle.com>

* origin/main: (156 commits) Upgrade Netris SDK version to 1.1.0 since 24.0 (apache#14365) feature: instance boot group and readiness rules (apache#13907) dont show unathorized notification for prelogin (apache#14054) .github: Upgrade docker/login-action to fix ASF Allowlist failure (apache#14276) virtual router: Add route-maps to BGP peers for Routed Mode (apache#9964) Add pre-commit hook to validate dependabot.yml (apache#14107) Add the ASF Allowlist check workflow (apache#14113) Revert "VMware to KVM: cold (VDDK) and warm (CBT) migration into Ceph/RBD and Linstor, plus importVm adoption of existing RBD root volumes and Linstor root/data volumes (apache#13656)" VMware to KVM: cold (VDDK) and warm (CBT) migration into Ceph/RBD and Linstor, plus importVm adoption of existing RBD root volumes and Linstor root/data volumes (apache#13656) engine-schema: upgrade path for 24.0.0 (apache#14033) xenserver/xcpng: do not bypass secondary storage when copy volumes between pools (apache#13750) backport: Incremental NAS backup support for KVM (apache#13074) (apache#13877) storage: KVM - enable RBD/Ceph volume encryption support (apache#13556) backup: remove the powered-off precondition for all backup providers (apache#14094) Nas backup: Fix mount/unmount error handling and timeout in LibvirtRestoreBackupCommandWrapper (apache#14006) kvm: apply rbd_default_data_pool when creating volumes from templates on RBD (apache#13361) server: copy template details to vm instance details when import VM (apache#12793) marvin: use pycryptodome instead of PyCrypt (apache#13594) Packaging: accept normalized Marvin archive names (apache#13793) SharedFS: fix NPE when accessing systemvm templates for unsupported hypervisors (apache#13930) ...

clayburn · 2026-10-12T10:23:53Z

Hi @DaanHoogland and @GutoVeronezi, it was nice meeting you today. This is the open PR for connecting CloudStack to Develocity as discussed.

Note there is one action required: please open a ticket with ASF Infra to request access to the GitHub Actions secret to publish Build Scans to Develocity. They will be able to add your repo to the allowlist, which will then allow Build Scan publication to work.

Copilot

Warning

Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.

Copilot review overview

3 open findings

The expression syntax here (#{{...}}) is inconsistent with the other Develocity expressions in… · New
With <enabled>false</enabled>, the remote build cache is disabled, so storeEnabled is either… · New
DEVELOCITY_ACCESS_KEY is being repeated across multiple steps/workflows. To reduce duplication… · New

7 resolved since last review

🧠 Review effort: Lite

Copilot

Warning

Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.

Copilot review overview

5 open findings

DEVELOCITY_ACCESS_KEY is being injected into steps that don't invoke Maven (including a step that… · New
DEVELOCITY_ACCESS_KEY is being injected into steps that don't invoke Maven (including a step that… · New
DEVELOCITY_ACCESS_KEY is being injected into steps that don't invoke Maven (including a step that… · New
The expression syntax #{{'0.0.0.0'}} is likely invalid for Develocity/Maven configuration… · New
With <enabled>false</enabled>, the remote build cache is disabled, so <storeEnabled> will never… · New

3 resolved since last review

🧠 Review effort: Lite

      - name: Setup Simulator Prerequisites
+        env:
+          DEVELOCITY_ACCESS_KEY: ${{ secrets.DEVELOCITY_ACCESS_KEY }}


      - name: Start CloudStack Management Server with Simulator
+        env:
+          DEVELOCITY_ACCESS_KEY: ${{ secrets.DEVELOCITY_ACCESS_KEY }}


      - name: Run Integration Tests with Simulator
+        env:
+          DEVELOCITY_ACCESS_KEY: ${{ secrets.DEVELOCITY_ACCESS_KEY }}


+  <buildScan>
+    <backgroundBuildScanUpload>#{isFalse(env['GITHUB_ACTIONS'])}</backgroundBuildScanUpload>
+    <obfuscation>
+      <ipAddresses>#{{'0.0.0.0'}}</ipAddresses>


+    </local>
+    <remote>
+      <enabled>false</enabled>
+      <storeEnabled>#{isTrue(env['GITHUB_ACTIONS'])}</storeEnabled>


DaanHoogland · 2026-10-13T09:08:22Z

Hi @DaanHoogland and @GutoVeronezi, it was nice meeting you today. This is the open PR for connecting CloudStack to Develocity as discussed.

Note there is one action required: please open a ticket with ASF Infra to request access to the GitHub Actions secret to publish Build Scans to Develocity. They will be able to add your repo to the allowlist, which will then allow Build Scan publication to work.

FYI: https://issues.apache.org/jira/browse/INFRA-28516

DaanHoogland · 2026-10-13T11:42:26Z

Hi @DaanHoogland and @GutoVeronezi, it was nice meeting you today. This is the open PR for connecting CloudStack to Develocity as discussed.
Note there is one action required: please open a ticket with ASF Infra to request access to the GitHub Actions secret to publish Build Scans to Develocity. They will be able to add your repo to the allowlist, which will then allow Build Scan publication to work.

FYI: https://issues.apache.org/jira/browse/INFRA-28516

@GutoVeronezi , the token is created by infra, can you review/merge? DEVELOCITY_ACCESS_KEY is created.

clayburn · 2026-10-13T12:10:21Z

@GutoVeronezi - please note in your testing that this PR cannot access the secret as it is from a fork. You would need to push my branch to apache/cloudstack to properly validate

GutoVeronezi · 2026-10-13T13:38:08Z

@GutoVeronezi - please note in your testing that this PR cannot access the secret as it is from a fork. You would need to push my branch to apache/cloudstack to properly validate

Hello @clayburn, I just created PR #14394 with the changes in the official repo

GutoVeronezi · 2026-10-13T13:42:21Z

Closing in favor of #14394

ribafish added 2 commits June 3, 2024 22:51

Added basic Develocity integration

bf508af

Added develocity access key setup to github actions (maven) workflows

e94e428

boring-cyborg Bot added the component:integration label Jun 3, 2024

Cleanup

2601328

ribafish force-pushed the gk/develocityIntegration branch from a565234 to 2601328 Compare June 4, 2024 10:11

ribafish added 2 commits June 4, 2024 16:15

Added licensing header to .mvn/extensions.xml

217cd5d

Added lincesing info and .gitignore for develocity integration

9c66ca4

weizhouapache approved these changes Jun 10, 2024

View reviewed changes

DaanHoogland approved these changes Aug 16, 2026

View reviewed changes

DaanHoogland added this to CloudStack Testing Aug 31, 2026

github-project-automation Bot moved this to Backlog in CloudStack Testing Aug 31, 2026

DaanHoogland moved this from Backlog to Ready in CloudStack Testing Aug 31, 2026

DaanHoogland assigned RosiKyu Sep 1, 2026

Update develocity-maven-extension version to 2.6.0

e8d7b4c

Copilot AI lite review requested due to automatic review settings October 12, 2026 09:22

Copilot AI reviewed Oct 12, 2026

View reviewed changes

Comment thread .mvn/develocity.xml

Comment thread .github/workflows/ci.yml Outdated

Comment thread .github/workflows/build.yml Outdated

Copilot AI lite review requested due to automatic review settings October 12, 2026 09:39

Copilot stopped reviewing on behalf of ribafish due to an error October 12, 2026 09:40
An unexpected error occurred. For more details, see the detailed logs in GitHub Actions.

Copilot AI reviewed Oct 12, 2026

View reviewed changes

Comment thread .mvn/develocity.xml

Comment thread .github/workflows/ci.yml Outdated

Comment thread .mvn/develocity.xml

Comment thread .mvn/develocity.xml

Copilot stopped reviewing on behalf of ribafish due to an error October 12, 2026 10:03
An unexpected error occurred. For more details, see the detailed logs in GitHub Actions.

ribafish added 2 commits October 12, 2026 12:09

Scope DEVELOCITY_ACCESS_KEY to the Maven steps

d64adb4

Signed-off-by: Gasper Kojek <gkojek@gradle.com>

Copilot AI lite review requested due to automatic review settings October 12, 2026 10:11

Copilot AI reviewed Oct 12, 2026

View reviewed changes

Comment thread .mvn/develocity.xml

Comment thread .mvn/develocity.xml

Comment thread .github/workflows/ci.yml

Copilot stopped reviewing on behalf of ribafish due to an error October 12, 2026 11:18
An unexpected error occurred. For more details, see the detailed logs in GitHub Actions.

DaanHoogland requested a lite review from Copilot October 12, 2026 15:31

Copilot AI reviewed Oct 12, 2026

View reviewed changes

Copilot stopped reviewing on behalf of DaanHoogland due to an error October 12, 2026 16:57
The job was not acquired by a runner even after multiple attempts.

GutoVeronezi mentioned this pull request Oct 13, 2026

Develocity integration #14394

Open

GutoVeronezi closed this Oct 13, 2026

kiranchavala removed this from the 24.0 milestone Oct 13, 2026

Conversation

ribafish commented Jun 3, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Types of changes

Uh oh!

boring-cyborg Bot commented Jun 3, 2024

Uh oh!

codecov Bot commented Jun 4, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

GutoVeronezi commented Jun 5, 2024

Uh oh!

GutoVeronezi commented Jun 5, 2024

Uh oh!

blueorangutan commented Jun 5, 2024

Uh oh!

blueorangutan commented Jun 5, 2024

Uh oh!

DaanHoogland commented Jun 6, 2024

Uh oh!

GutoVeronezi commented Jun 7, 2024

Uh oh!

DaanHoogland commented Jun 7, 2024

Uh oh!

GutoVeronezi commented Jun 7, 2024

Uh oh!

vishesh92 commented Jun 8, 2024

Uh oh!

weizhouapache commented Jun 8, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ribafish commented Jun 10, 2024

Uh oh!

weizhouapache commented Jun 10, 2024

Uh oh!

ribafish commented Jun 10, 2024

Uh oh!

weizhouapache commented Jun 10, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

weizhouapache left a comment

Choose a reason for hiding this comment

Uh oh!

weizhouapache commented Jun 10, 2024

Uh oh!

clayburn commented Jun 10, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

weizhouapache commented Jun 10, 2024

Uh oh!

clayburn commented Jun 10, 2024

Uh oh!

DaanHoogland commented Jun 11, 2024

Uh oh!

blueorangutan commented Jun 11, 2024

Uh oh!

DaanHoogland commented Aug 16, 2026

Uh oh!

DaanHoogland left a comment

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Copilot review overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Copilot review overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

clayburn commented Oct 12, 2026

Uh oh!

ribafish commented Jun 3, 2024 •

edited

Loading

codecov Bot commented Jun 4, 2024 •

edited

Loading

weizhouapache commented Jun 8, 2024 •

edited

Loading

weizhouapache commented Jun 10, 2024 •

edited

Loading

clayburn commented Jun 10, 2024 •

edited

Loading

GutoVeronezi commented Oct 13, 2026 •

edited

Loading