AWS S3 CRT Transfer Manager uses an ever-increasing amount of memory based on the number of objects to sync #10880

Describe the bug

When performing an AWS S3 Sync operation with a very large number of files (millions) there appears to be an slow but unbounded increase in memory usage until the transfer completes when using the CRT transfer manager. This does not occur when using the classic client.

This appears to be from the _future_coordinators variable here:

This variable is appended to, but never deleted from until the transfer is cancelled or the sync is fully done.

Regression Issue

• Select this option if this issue appears to be a regression.

Expected Behavior

AWS S3 Sync commands on very large number of objects should not use an ever-increasing amount of memory.

Current Behavior

AWS S3 Sync commands accumulate memory usage until the command is complete.

Reproduction Steps

Run an aws s3 sync command on a very large number of files (e.g., millions). Observe that memory usage will continue to increase until the command completes. For ease of reproduction, this works in an AWS emulator such as Ministack or LocalStack as well, you don't need to sync to an actual AWS bucket.

Possible Solution

Clean up entries from _future_coordinators as uploads complete

Additional Information/Context

No response

CLI version used

2.38.3

Environment details (OS name and version, etc.)

aws-cli/2.38.3 Python/3.15.0 Darwin/25.6.0 source/arm64