Request carrier-scoped ddprof context storage on JDK 21+ by rkennke · Pull Request #11826 · DataDog/dd-trace-java

rkennke · 2026-07-01T11:23:06Z

What Does This Do

On JDK 21+, before the ddprof profiler is loaded, this exports java.base/jdk.internal.misc to the classloader that loads com.datadoghq.profiler.*.

Implemented as a small prepareDatadogProfilerContextStorage(Instrumentation) helper in Agent, invoked in InstallDatadogTracerCallback right before installDatadogTracer (which triggers the profiler load, so the export is in place before JavaProfiler is constructed).

Motivation

The profiler exposes its OTEL context ThreadContext as a DirectByteBuffer over the carrier thread's native record — the record the (carrier-bound) sampler reads. When that storage is keyed by the virtual thread (the current default), a mounted vthread pins to whichever carrier it first ran on:

Wrong after migration — writes land on the old carrier, so a sampler on the new carrier sees stale/empty context; and
Unsafe once the old carrier's OS thread exits — the native record is freed while the buffer keeps being written, a use-after-free that can corrupt JVM-owned native memory (seen in the field as a SIGSEGV in ThreadsSMRSupport::free_list under a Loom workload).

The java-profiler fix (PROF-15271) adds carrier-scoped storage via CarrierThreadLocal, so a mounted vthread always resolves to its current carrier's live record. CarrierThreadLocal is in a non-exported package, hence the export.

Additional Notes

Only the export lives here; the storage-mode switch stays internal to java-profiler. An earlier revision of this PR also set the profiler's ddprof.debug.context.storage.mode system property from tracer code. Per review, that's an internal ddprof knob that may still change and shouldn't be entangled with the generic Agent class; the profiler's own version bump (already queued) is what activates carrier mode. This PR is now scoped to just the export, which is harmless to ship regardless of which ddprof version is bundled.
Guarded by isProfilingEnabled() && isDatadogProfilerEnabled() && !isWindows() && isJavaVersionAtLeast(21). JDK9ModuleAccess (internal-api-9) is only referenced under the 21+ guard, so it never loads on Java 8.
Exporting jdk.internal.misc via redefineModule follows existing precedent (AdvancedAgentChecks does the same for CDS detection).

Contributor Checklist

Title formatted per contribution guidelines
Assign type: / comp: labels
No close/fix linking keywords used

Jira ticket: PROF-15271

🤖 Generated with Claude Code

…PROF-15271) The ddprof profiler exposes its OTEL context ThreadContext as a DirectByteBuffer over the carrier thread's native record. When that storage is keyed by the virtual thread (the pre-fix default), a mounted vthread pins to its first carrier: writes land on the wrong carrier after migration, and become a use-after-free once that carrier's OS thread exits. The java-profiler fix (PROF-15271) adds carrier-scoped storage via jdk.internal.misc.CarrierThreadLocal, selected by the ddprof.context.storage.mode system property. Before the profiler is loaded, on JDK 21+ (profiler enabled, non-Windows): - export java.base/jdk.internal.misc to the classloader that loads com.datadoghq.profiler.* so CarrierThreadLocal is reachable, and - set ddprof.context.storage.mode=carrier (only if unset), which requests carrier scoping and makes the profiler fail fast if it cannot honor it. Both actions are inert against ddprof builds that predate carrier support (the property is ignored; the export goes unused), so this is safe to ship ahead of the ddprof version bump — it activates automatically when that lands. Operators opt out with -Dddprof.context.storage.mode=thread (or =auto); we only set the property when unset, so an explicit choice always wins. No separate dd.* config key: the profiler's own system property is the single kill-switch. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

datadog-prod-us1-3 · 2026-07-01T11:31:51Z

🎯 Code Coverage (details)
• Patch Coverage: 0.00%
• Overall Coverage: 60.80% (+3.86%)

_{This comment will be updated automatically if new data arrives.

🔗 Commit SHA: 7ba6d03 | Docs | Datadog PR Page | Give us feedback!}

dd-octo-sts · 2026-07-01T11:39:24Z

🟡 Java Benchmark SLOs — Performance SLO warning (near threshold)

Suite	Status
Startup	🟡 warning

SLO thresholds are defined here based on automatically generated metrics. A warning is raised when results are within 5% of the threshold.

PR vs. master results

Scenario	Candidate	master	Δ (95% CI of mean)
startup:insecure-bank:iast:Agent	14.06 s	13.96 s	[+0.1%; +1.4%] (maybe worse)
startup:insecure-bank:tracing:Agent	12.91 s	13.03 s	[-1.7%; -0.1%] (maybe better)
startup:petclinic:appsec:Agent	17.00 s	16.76 s	[+0.4%; +2.5%] (maybe worse)
startup:petclinic:iast:Agent	16.89 s	16.28 s	[-0.6%; +8.1%] (no difference)
startup:petclinic:profiling:Agent	16.82 s	16.93 s	[-1.7%; +0.4%] (no difference)
startup:petclinic:sca:Agent	16.96 s	16.77 s	[+0.1%; +2.2%] (maybe worse)
startup:petclinic:tracing:Agent	16.20 s	16.07 s	[-0.0%; +1.7%] (no difference)

Commit: 7ba6d033 · CI Pipeline · Benchmarking Platform UI

Load and DaCapo benchmarks can be triggered manually in the GitLab pipeline. Results will appear in the Benchmarking Platform UI after completion.

…dogTracer Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

…ROF-15271) The java-profiler side renamed the selector to ddprof.debug.context.storage.mode (ddprof.debug.*, signalling an internal knob). Match the property name we export and set here so the carrier request still takes effect. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

jbachorik

In general, looking good.

My only concerns are about not entangling the generic Agent with profiler specific configs and quirks.

…he export Per review, don't set the profiler's internal ddprof.debug.context.storage.mode property from tracer code -- it's an implementation detail that may change and the profiler bump activating carrier mode is already in the merge queue. The remaining change is exporting java.base/jdk.internal.misc so java-profiler can reach CarrierThreadLocal.

rkennke · 2026-07-06T17:34:26Z

/merge

gh-worker-devflow-routing-ef8351 · 2026-07-06T17:34:30Z

View all feedbacks in Devflow UI.

2026-07-06 17:34:29 UTC ℹ️ Start processing command /merge

2026-07-06 17:34:39 UTC ℹ️ MergeQueue: waiting for PR to be ready

This pull request is not mergeable according to GitHub. Common reasons include pending required checks, missing approvals, or merge conflicts — but it could also be blocked by other repository rules or settings.
It will be added to the queue as soon as checks pass and/or get approvals. View in MergeQueue UI.
Note: if you pushed new commits since the last approval, you may need additional approval.
You can remove it from the waiting list with /remove command.

2026-07-06 19:09:11 UTC ℹ️ MergeQueue: merge request added to the queue

The expected merge time in master is approximately 1h (p90).

2026-07-06 20:16:01 UTC ℹ️ MergeQueue: This merge request was merged

…11826) profiling: request carrier-scoped ddprof context storage on JDK 21+ (PROF-15271) The ddprof profiler exposes its OTEL context ThreadContext as a DirectByteBuffer over the carrier thread's native record. When that storage is keyed by the virtual thread (the pre-fix default), a mounted vthread pins to its first carrier: writes land on the wrong carrier after migration, and become a use-after-free once that carrier's OS thread exits. The java-profiler fix (PROF-15271) adds carrier-scoped storage via jdk.internal.misc.CarrierThreadLocal, selected by the ddprof.context.storage.mode system property. Before the profiler is loaded, on JDK 21+ (profiler enabled, non-Windows): - export java.base/jdk.internal.misc to the classloader that loads com.datadoghq.profiler.* so CarrierThreadLocal is reachable, and - set ddprof.context.storage.mode=carrier (only if unset), which requests carrier scoping and makes the profiler fail fast if it cannot honor it. Both actions are inert against ddprof builds that predate carrier support (the property is ignored; the export goes unused), so this is safe to ship ahead of the ddprof version bump — it activates automatically when that lands. Operators opt out with -Dddprof.context.storage.mode=thread (or =auto); we only set the property when unset, so an explicit choice always wins. No separate dd.* config key: the profiler's own system property is the single kill-switch. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> docs: clarify profiler context-storage helper runs before installDatadogTracer Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> profiling: align ddprof context-storage property to ddprof.debug.* (PROF-15271) The java-profiler side renamed the selector to ddprof.debug.context.storage.mode (ddprof.debug.*, signalling an internal knob). Match the property name we export and set here so the carrier request still takes effect. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Merge branch 'master' into rkennke/PROF-15271 profiling: drop internal ddprof context-storage property, keep only the export Per review, don't set the profiler's internal ddprof.debug.context.storage.mode property from tracer code -- it's an implementation detail that may change and the profiler bump activating carrier mode is already in the merge queue. The remaining change is exporting java.base/jdk.internal.misc so java-profiler can reach CarrierThreadLocal. [ci: NON_DEFAULT_JVMS] trigger graalvm smoke tests Merge branch 'master' into rkennke/PROF-15271 Merge branch 'master' into rkennke/PROF-15271 Co-authored-by: jbachorik <jaroslav.bachorik@datadoghq.com> Co-authored-by: roman.kennke <roman.kennke@datadoghq.com>

rkennke added comp: profiling Profiling type: bug fix Bug fix labels Jul 1, 2026

docs: clarify profiler context-storage helper runs before installData…

21833f1

…dogTracer Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

rkennke added the tag: no release notes Changes to exclude from release notes label Jul 1, 2026

rkennke marked this pull request as ready for review July 1, 2026 12:18

rkennke requested a review from a team as a code owner July 1, 2026 12:18

rkennke requested a review from ygree July 1, 2026 12:18

dd-octo-sts Bot added the tag: ai generated Largely based on code generated by an AI or LLM label Jul 1, 2026

rkennke mentioned this pull request Jul 1, 2026

Profiling casuing JVM to crash #11749

Open

rkennke and others added 2 commits July 1, 2026 16:10

Merge branch 'master' into rkennke/PROF-15271

1b65abc

rkennke requested a review from jbachorik July 6, 2026 09:21

jbachorik reviewed Jul 6, 2026

View reviewed changes

Comment thread dd-java-agent/agent-bootstrap/src/main/java/datadog/trace/bootstrap/Agent.java Outdated

jbachorik reviewed Jul 6, 2026

View reviewed changes

Comment thread dd-java-agent/agent-bootstrap/src/main/java/datadog/trace/bootstrap/Agent.java Outdated

jbachorik reviewed Jul 6, 2026

View reviewed changes

Comment thread dd-java-agent/agent-bootstrap/src/main/java/datadog/trace/bootstrap/Agent.java

jbachorik reviewed Jul 6, 2026

View reviewed changes

Comment thread dd-java-agent/agent-bootstrap/src/main/java/datadog/trace/bootstrap/Agent.java Outdated

jbachorik requested changes Jul 6, 2026

View reviewed changes

rkennke added 2 commits July 6, 2026 14:18

[ci: NON_DEFAULT_JVMS] trigger graalvm smoke tests

a2e5507

jbachorik approved these changes Jul 6, 2026

View reviewed changes

jbachorik added tag: ai generated Largely based on code generated by an AI or LLM and removed tag: ai generated Largely based on code generated by an AI or LLM tag: no release notes Changes to exclude from release notes labels Jul 6, 2026

Merge branch 'master' into rkennke/PROF-15271

985013e

Merge branch 'master' into rkennke/PROF-15271

7ba6d03

gh-worker-dd-mergequeue-cf854d Bot merged commit 403fe7f into master Jul 6, 2026
587 checks passed

gh-worker-dd-mergequeue-cf854d Bot deleted the rkennke/PROF-15271 branch July 6, 2026 20:15

github-actions Bot added this to the 1.64.0 milestone Jul 6, 2026

PerfectSlayer changed the title ~~profiling: request carrier-scoped ddprof context storage on JDK 21+~~ Request carrier-scoped ddprof context storage on JDK 21+ Jul 7, 2026

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Request carrier-scoped ddprof context storage on JDK 21+#11826

Request carrier-scoped ddprof context storage on JDK 21+#11826
gh-worker-dd-mergequeue-cf854d[bot] merged 8 commits into
masterfrom
rkennke/PROF-15271

rkennke commented Jul 1, 2026 •

edited

Loading

Uh oh!

datadog-prod-us1-3 Bot commented Jul 1, 2026 •

edited by datadog-official Bot

Loading

Uh oh!

dd-octo-sts Bot commented Jul 1, 2026 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jbachorik left a comment

Uh oh!

rkennke commented Jul 6, 2026

Uh oh!

gh-worker-devflow-routing-ef8351 Bot commented Jul 6, 2026 •

edited

Loading

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

rkennke commented Jul 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What Does This Do

Motivation

Additional Notes

Contributor Checklist

Uh oh!

datadog-prod-us1-3 Bot commented Jul 1, 2026 • edited by datadog-official Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dd-octo-sts Bot commented Jul 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🟡 Java Benchmark SLOs — Performance SLO warning (near threshold)

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jbachorik left a comment

Choose a reason for hiding this comment

Uh oh!

rkennke commented Jul 6, 2026

Uh oh!

gh-worker-devflow-routing-ef8351 Bot commented Jul 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

rkennke commented Jul 1, 2026 •

edited

Loading

datadog-prod-us1-3 Bot commented Jul 1, 2026 •

edited by datadog-official Bot

Loading

dd-octo-sts Bot commented Jul 1, 2026 •

edited

Loading

gh-worker-devflow-routing-ef8351 Bot commented Jul 6, 2026 •

edited

Loading