Skip to content

Add GC support for external memory handles - #133986

Open
jkoritzinsky wants to merge 11 commits into
dotnet:mainfrom
jkoritzinsky:external-roots
Open

jkoritzinsky wants to merge 11 commits into
dotnet:mainfrom
jkoritzinsky:external-roots

Conversation

@jkoritzinsky

Copy link
Copy Markdown
Member

Summary

  • add AppDomain-managed external memory handles for GC-scanning managed references stored outside the GC heap and managed stacks
  • enumerate external memory handles through DAC and a dedicated cDAC ExternalMemoryHandles contract, including byref-like field walking and interior-pointer resolution
  • protect unboxed byref-like func-eval results for the lifetime of the returned ICorDebugValue

Testing

  • build.cmd tools+tools.cdactests -test
  • build.cmd clr

Note

This pull request description was generated with GitHub Copilot.

jkoritzinsky and others added 5 commits September 14, 2026 14:44
Transfer external memory handle ownership to the returned debugger value so byref-like result storage remains GC-protected for its lifetime.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot AI lite review requested due to automatic review settings September 15, 2026 19:03
@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines:
Successfully started running 5 pipeline(s).
11 pipeline(s) were filtered out due to trigger conditions.
There may be pipelines that require an authorized user to comment /azp run to run.

@dotnet-policy-service

Copy link
Copy Markdown
Contributor

Tagging subscribers to this area: @steveisok, @tommcdon, @dotnet/dotnet-diag
See info in area-owners.md if you want to be subscribed.

@jkoritzinsky
jkoritzinsky requested review from a team, davidwrighton and jkotas September 15, 2026 19:05

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟡 Changes recommended

Unresolved critical and moderate findings affect func-eval handling, disposal failure paths, and cDAC root classification.

Get a fresh assessment by requesting another Copilot review.

Review tier: Lite
Findings: 1 High severity

Open (1)
What changed in this PR

Adds AppDomain-managed external memory handles so GC, DAC/cDAC, and debugger func-eval can track managed references outside the GC heap.

Changes:

  • Adds runtime handle storage, GC scanning, DAC/cDAC enumeration, and pointer resolution.
  • Extends cDAC contracts, RefWalk support, and inline-array metadata.
  • Protects byref-like func-eval results and adds tests/documentation.
File Summary
src/​native/​managed/​cdac/​tests/​UnitTests/​RefWalkExternalMemoryHandleTests.cs Tests external-root discovery and byref-like walking.
src/​native/​managed/​cdac/​tests/​UnitTests/​RefWalkExternalMemoryHandlesContractTests.cs Tests RefWalk contract integration.
src/​native/​managed/​cdac/​tests/​UnitTests/​MethodTableTests.cs Tests inline-array metadata.
src/​native/​managed/​cdac/​tests/​UnitTests/​ExternalMemoryHandlesTests.cs Tests external-handle enumeration.
src/​native/​managed/​cdac/​tests/​UnitTests/​ContractDescriptor/​TargetTests.cs Tests contract descriptor coverage.
src/​native/​managed/​cdac/​tests/​TestInfrastructure/​TestPlaceholderTarget.cs Provides test placeholder target infrastructure.
src/​native/​managed/​cdac/​tests/​DumpTests/​DacDbi/​DacDbiRefWalkDumpTests.cs Tests dump-based RefWalk behavior.
src/​native/​managed/​cdac/​Microsoft.Diagnostics.DataContractReader.Legacy/​Dbi/​Helpers/​RefWalk.cs Integrates external roots with legacy RefWalk.
src/​native/​managed/​cdac/​Microsoft.Diagnostics.DataContractReader.Contracts/​DataType.cs Registers the external-handle data type.
src/​native/​managed/​cdac/​Microsoft.Diagnostics.DataContractReader.Contracts/​Data/​ExternalMemoryHandle.cs Defines external-handle data.
src/​native/​managed/​cdac/​Microsoft.Diagnostics.DataContractReader.Contracts/​Data/​EEClass.cs Reads inline-array metadata.
src/​native/​managed/​cdac/​Microsoft.Diagnostics.DataContractReader.Contracts/​Data/​AppDomain.cs Describes AppDomain handle data.
src/​native/​managed/​cdac/​Microsoft.Diagnostics.DataContractReader.Contracts/​CoreCLRContracts.cs Registers and validates contracts.
src/​native/​managed/​cdac/​Microsoft.Diagnostics.DataContractReader.Contracts/​Contracts/​StackWalk/​GC/​GcScanContext.cs Uses shared GC pointer-resolution support.
src/​native/​managed/​cdac/​Microsoft.Diagnostics.DataContractReader.Contracts/​Contracts/​StackWalk/​GC/​GCInteriorPointerResolver.cs Resolves interior pointers.
src/​native/​managed/​cdac/​Microsoft.Diagnostics.DataContractReader.Contracts/​Contracts/​RuntimeTypeSystem_1.cs Implements inline-array inspection.
src/​native/​managed/​cdac/​Microsoft.Diagnostics.DataContractReader.Contracts/​Contracts/​ExternalMemoryHandles_1.cs Enumerates external roots. Moderate: checks GC flags for interior roots incorrectly.
src/​native/​managed/​cdac/​Microsoft.Diagnostics.DataContractReader.Abstractions/​Contracts/​IRuntimeTypeSystem.cs Adds inline-array type inspection.
src/​native/​managed/​cdac/​Microsoft.Diagnostics.DataContractReader.Abstractions/​Contracts/​IExternalMemoryHandles.cs Defines the external-handle API. Nit: lacks a linked API approval issue.
src/​native/​managed/​cdac/​Microsoft.Diagnostics.DataContractReader.Abstractions/​ContractRegistry.cs Registers external-memory-handle contract access.
src/​coreclr/​vm/​gcenv.ee.cpp Integrates external handles into GC root scanning.
src/​coreclr/​vm/​externalmemoryhandle.h Defines external-handle structures.
src/​coreclr/​vm/​externalmemoryhandle.cpp Implements external-handle scanning.
src/​coreclr/​vm/​datadescriptor/​datadescriptor.inc Adds runtime data descriptors.
src/​coreclr/​vm/​CMakeLists.txt Includes the new native source in builds.
src/​coreclr/​vm/​appdomain.hpp Declares AppDomain handle storage and APIs.
src/​coreclr/​vm/​appdomain.cpp Manages, scans, and dumps external handles.
src/​coreclr/​inc/​CrstTypes.def Defines the external-handle lock.
src/​coreclr/​inc/​crsttypes_generated.h Provides generated lock metadata.
src/​coreclr/​debug/​inc/​dbgipceventtypes.h Defines debugger disposal event metadata.
src/​coreclr/​debug/​inc/​dbgipcevents.h Extends debugger IPC payloads.
src/​coreclr/​debug/​ee/​funceval.cpp Protects byref-like func-eval results. Critical: unboxed return data can still be treated as an object for strong-handle creation.
src/​coreclr/​debug/​ee/​debugger.h Declares debugger external-handle support.
src/​coreclr/​debug/​ee/​debugger.cpp Creates and disposes debugger handles.
src/​coreclr/​debug/​di/​rsthread.cpp Transfers and cleans up func-eval handles. Moderate: failed disposal events can leak the handle and buffer.
src/​coreclr/​debug/​di/​rspriv.h Stores debugger-side handle state.
src/​coreclr/​debug/​di/​process.cpp Processes handle disposal events.
src/​coreclr/​debug/​di/​divalue.cpp Associates handles with debugger values. Moderate: clears the token without safely handling IPC-send failure.
src/​coreclr/​debug/​daccess/​dacdbiimpl.h Declares DAC external-root state.
src/​coreclr/​debug/​daccess/​dacdbiimpl.cpp Enumerates external roots through DAC.
docs/​design/​datacontracts/​RuntimeTypeSystem.md Documents runtime type metadata.
docs/​design/​datacontracts/​ExternalMemoryHandles.md Documents the external-memory-handle contract.
docs/​design/​datacontracts/​data-descriptor-meanings.json Documents new descriptor meanings.

Comment thread src/coreclr/debug/ee/funceval.cpp
@jkotas
jkotas requested a review from a team September 15, 2026 19:24
Comment thread src/coreclr/inc/CrstTypes.def Outdated
@jkotas

jkotas commented Sep 15, 2026

Copy link
Copy Markdown
Member

Have you been able to observe debugger crashes or misbehaviors that are fixed by this change?

@janvorli

Copy link
Copy Markdown
Member

@jkoritzinsky I haven't heard about external memory handles before. What do we plan to use them for?

Comment thread docs/design/datacontracts/data-descriptor-meanings.json Outdated
@jkoritzinsky

Copy link
Copy Markdown
Member Author

Have you been able to observe debugger crashes or misbehaviors that are fixed by this change?

I have not observed any debugger crashes, but I did find in the func-eval code that we create a boxed byreflike on the heap to represent the result of the funceval:

if ((pDE->m_evalType != DB_IPCE_FET_NEW_OBJECT) && !RetValueType.IsNull())
{
ValidateFuncEvalReturnType(pDE->m_evalType, RetValueType.GetMethodTable());
RetValueType.GetMethodTable()->EnsureInstanceActive();
retObject = AllocateObject(RetValueType.GetMethodTable());

This is how we currently don't have a bunch of GC holes with funceval returns, but it's definitely not supported.

We block allocating byreflike values on the GC heap in all of the boxing paths:

// we shouldn't allow boxing of types that contains stack pointers
// csc and vbc already disallow it.
if (VMClsHnd.AsMethodTable()->IsByRefLike())
COMPlusThrow(kInvalidProgramException,W("NotSupported_ByRefLike"));

// Don't allow ref structs
if (!allowByRefLike && pMT->IsByRefLike())
{
COMPlusThrow(kNotSupportedException, W("NotSupported_ByRefLike"));
}

if (IsByRefLike())
{
// We should never box a type that contains stack pointers.
COMPlusThrow(kInvalidOperationException, W("InvalidOperation_TypeCannotBeBoxed"));
}

However, we don't block it in AllocateObject itself:

// AllocateObject will throw OutOfMemoryException so don't need to check
// for NULL return value from it.
OBJECTREF AllocateObject(MethodTable *pMT
, GC_ALLOC_FLAGS flags
#ifdef FEATURE_COMINTEROP
, bool fHandleCom
#endif
)
{
CONTRACTL {
THROWS;
GC_TRIGGERS;
MODE_COOPERATIVE; // returns an objref without pinning it => cooperative
PRECONDITION(CheckPointer(pMT));
PRECONDITION(pMT->CheckInstanceActivated());
} CONTRACTL_END;
// use unchecked oref here to avoid triggering assert in Validate that the AD is
// not set becuase it isn't until near the end of the fcn at which point we can allow
// the check.
_UNCHECKED_OBJECTREF oref;
SetTypeHandleOnThreadForAlloc(TypeHandle(pMT));
#ifdef FEATURE_COMINTEROP
#ifdef FEATURE_COMINTEROP_UNMANAGED_ACTIVATION
if (fHandleCom && pMT->IsComObjectType())
{
if (!g_pConfig->IsBuiltInCOMSupported())
{
COMPlusThrow(kNotSupportedException, W("NotSupported_COM"));
}
// Create a instance of __ComObject here is not allowed as we don't know what COM object to create
if (pMT == g_pBaseCOMObject)
COMPlusThrow(kInvalidComObjectException, IDS_EE_NO_BACKING_CLASS_FACTORY);
OBJECTREF obj = AllocateComObject_ForManaged(pMT);
oref = OBJECTREF_TO_UNCHECKED_OBJECTREF(obj);
}
#endif // FEATURE_COMINTEROP_UNMANAGED_ACTIVATION
#else // FEATURE_COMINTEROP
if (pMT->IsComObjectType())
{
COMPlusThrow(kPlatformNotSupportedException, IDS_EE_ERROR_COM);
}
#endif // FEATURE_COMINTEROP
else
{
if (pMT->ContainsGCPointers())
flags |= GC_ALLOC_CONTAINS_REF;
if (pMT->HasFinalizer())
flags |= GC_ALLOC_FINALIZE;
DWORD totalSize = pMT->GetBaseSize();
if (totalSize >= LARGE_OBJECT_SIZE && totalSize >= GCHeapUtilities::GetGCHeap()->GetLOHThreshold())
flags |= GC_ALLOC_LARGE_OBJECT_HEAP;
#ifdef FEATURE_64BIT_ALIGNMENT
if (pMT->RequiresAlign8())
{
// The last argument to the allocation, indicates whether the alignment should be "biased". This
// means that the object is allocated so that its header lies exactly between two 8-byte
// boundaries. This is required in cases where we need to mis-align the header in order to align
// the actual payload. Currently this is false for classes (where we apply padding to ensure the
// first field is aligned relative to the header) and true for boxed value types (where we can't
// do the same padding without introducing more complexity in type layout and unboxing stubs).
_ASSERTE(sizeof(Object) == 4);
flags |= GC_ALLOC_ALIGN8;
if (pMT->IsValueType())
flags |= GC_ALLOC_ALIGN8_BIAS;
}
#endif // FEATURE_64BIT_ALIGNMENT
Object* orObject = (Object*)Alloc(totalSize, flags);
if (flags & GC_ALLOC_USER_OLD_HEAP)
{
orObject->SetMethodTableForUOHObject(pMT);
}
else
{
orObject->SetMethodTable(pMT);
}
PublishObjectAndNotify(orObject, flags);
oref = OBJECTREF_TO_UNCHECKED_OBJECTREF(orObject);
}
return UNCHECKED_OBJECTREF_TO_OBJECTREF(oref);
}

This PR gets us off this unsupported path.

I haven't heard about external memory handles before. What do we plan to use them for?

I plan to use them basically for supporting func-eval (this PR allocates a handle for the return of a funceval). In a future PR, I want to use them to replace the func-eval use cases for ProtectValueClassFrame as well (byreflike args that can't be boxed), enabling me to close #133730 in favor of this PR. The other usages of ProtectValueClassFrame today can all accept boxing (RuntimeMethodInfo.Invoke) so those cases will just move to the boxing path.

CCR flagged that the strong-handle path treated any non-null RetValueType
as a boxed object, including byref-like results whose slot holds value
data copied into the external memory handle rather than an OBJECTREF.
Exclude byref-like RetValueType from that condition; the external
memory handle remains the sole GC root for byref-like func-eval results.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot AI review requested due to automatic review settings September 15, 2026 21:17
The new lock has no dependencies with other Crst types, so it can
participate in ranking instead of being explicitly unordered.
Regenerated crsttypes_generated.h via CrstTypeTool.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🔵 Needs a closer look

End-to-end CoreCLR coverage is still requested for real external-handle GC scanning and retention.

Review tier: Lite
Findings: None

Resolved since last review (1)

Copilot AI review requested due to automatic review settings September 15, 2026 21:37

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🔵 Needs a closer look

Cleanup and partial-dump handling issues remain, end-to-end func-eval lifetime coverage is missing, and public API approval is unresolved.

Review tier: Lite
Findings: None

Previously missed (1)

In code that hasn't changed since last review

Medium severity Add an end-to-end debugger lifetime regression test

src/​coreclr/​debug/​ee/​funceval.cpp:2821

These changes allocate/register a real GC root and transfer ownership across the debugger IPC boundary, but the added tests only use mocked cDAC targets; none exercises actual FuncEvalComplete/GC/ICorDebugValue lifetime or cleanup after GetResult. Add an end-to-end debugger regression that forces a collection while the returned byref-like value is held and verifies disposal after release; this is the behavior the native changes are intended to protect.

[!NOTE] This review comment was generated by GitHub Copilot.

The external memory handle list was previously a member of AppDomain,
even though there is only ever a single AppDomain per process. Move
the list (and its lock) to a static on ExternalMemoryHandle itself so
the type owns its own lifetime and stops hanging off AppDomain's
grab-bag of unrelated state.

- AppDomain::AddExternalMemoryHandle/RemoveExternalMemoryHandle/
  GCScanExternalMemoryHandles become ExternalMemoryHandle::Add/Remove/
  GCScanRoots (plus a new Init/Cleanup pair called from
  AppDomain::Init/~AppDomain).
- DebuggerExternalMemoryHandle and DacRefWalker::WalkExternalMemoryHandles
  no longer need an AppDomain to reach the list.
- The cDAC ExternalMemoryHandles contract now reads a new
  ExternalMemoryHandles global pointer instead of walking through the
  AppDomain data descriptor.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot AI review requested due to automatic review settings September 15, 2026 22:31
jkoritzinsky and others added 2 commits September 15, 2026 15:45
CrstStatic (rather than CrstExplicitInit) is the established pattern for
a genuine process-wide static Crst that is explicitly Init()'d and never
destructed.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟡 Changes recommended

One or more issues must be addressed before approval.

Get a fresh assessment by requesting another Copilot review.

Review tier: Lite
Findings: 1 High severity

Open (1)

Comment thread src/coreclr/vm/externalmemoryhandle.cpp Outdated
Copilot AI review requested due to automatic review settings September 15, 2026 22:48

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟡 Changes recommended

Unresolved critical cleanup and moderate debugger/cDAC error-handling findings block approval.

Get a fresh assessment by requesting another Copilot review.

Review tier: Lite
Findings: 2 High severity

Open (2)

Comment on lines +80 to +83
while (!s_handles.IsEmpty())
{
delete s_handles.RemoveHead();
}
Add/Remove mutate the ExternalMemoryHandle list under s_crst, but
GCScanRoots walked it without taking that lock. During the initial,
concurrent mark phase of a background GC the EE keeps running, so a
debugger func-eval could concurrently insert or delete a node while the
GC thread iterates, risking a use-after-free (delete) or a missed root
(insert).

Follow the existing pattern used for interop ref-counted handle
callbacks (Interop::OnBeforeGCScanRoots/OnAfterGCScanRoots) and for
AppDomain statics (EnumAllStaticGCRefs), which skip work during the
concurrent phase: only invoke ExternalMemoryHandle::GCScanRoots when
sc->concurrent is false. The list can only be mutated while the EE is
suspended for a GC or by a thread in cooperative mode, so it is safe to
walk lock-free during the blocking mark/relocate phases and the final,
EE-suspended mark phase of a background GC; the concurrent mark phase
is simply skipped, and the subsequent final mark phase sees the
up-to-date list.
Copilot AI review requested due to automatic review settings September 15, 2026 23:09
Copilot stopped reviewing on behalf of jkoritzinsky due to an error September 15, 2026 23:30

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.


m_AssemblyCache.Clear();

ExternalMemoryHandle::Cleanup();
Comment on lines +85 to +91
private void AddInteriorRoot(
List<ExternalMemoryHandleRootData> roots,
TargetPointer slotAddress,
bool resolveInteriorPointers)
{
TargetPointer obj = _target.ReadPointer(slotAddress.Value);
if (obj == TargetPointer.Null || obj.Value == ulong.MaxValue)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

Status: No status

Development

Successfully merging this pull request may close these issues.

4 participants