Conversation
… assets New API layer for binary file storage that delegates to StoragePersistenceAPI. Registered in APILocator, defaults to FILE_SYSTEM mode with zero behavior change. Includes getBinaryFile, storeBinary (with hard-link support), copyBinary, deleteBinary, deleteAllBinaries, and existsBinary operations.
Migrate all 6 hardcoded filesystem binary access sites to use BinaryAssetStorageAPI. Deprecate getRealAssetPath methods with migration guidance. Add deleteAllBinaries for inode-level cleanup. Integration tests written for end-to-end verification. Migrated callers: - Contentlet.getBinary → BinaryAssetStorageAPI.getBinaryFile - ESContentletAPIImpl.handleBinaries → storeBinary/copyBinary - ESContentletAPIImpl.getBinaryFile → getBinaryFile - ContentletTransformer → getBinaryFile - ContentletJsonAPIImpl → getBinaryFile - WebAPI, FileAssetBundler, DropOldContentletRunner → getBinaryFile/deleteAllBinaries
… cache eviction Phase 3 of S3 binary asset storage: - Config-driven BINARY_CHAIN mode (FILE_SYSTEM → S3) with S3 init fallback - Binary S3 uses PathEncryptionMode.NONE for prefix-based listing - Added listObjectPaths(group, prefix) to StoragePersistenceAPI with FS/S3/Chain impls - getBinaryFile(inode, fieldVarName) discovers filename through chain on cold cache - deleteAllBinaries(inode) cleans all chain providers with partial-failure handling - BinaryCacheEvictionJob (StatefulJob) with LRU eviction, min-age protection - Registered eviction job in DotInitScheduler (cron-gated) - 10 new unit tests across storage providers and eviction job
Contributor
❌ Issue Linking RequiredThis PR could not be linked to an issue. All PRs must be linked to an issue for tracking purposes. How to fix this:Option 1: Add keyword to PR body (Recommended - auto-removes this comment) Edit this PR description and add one of these lines:
Option 2: Link via GitHub UI (Note: won't clear the failed check)
Option 3: Use branch naming Create a new branch with one of these patterns:
Why is this required?Issue linking ensures proper tracking, documentation, and helps maintain project history. It connects your code changes to the problem they solve. This comment was automatically generated by the issue linking workflow |
github-actions
Bot
added
Area : Backend
Contributor
|
Claude finished @swicken's task in 2m 3s —— View job
This is a draft POC with an explicit to-do list; I'm focusing on provable correctness/security/race findings in the new code, not re-flagging the acknowledged open gaps. |
swicken
added
the
PR: docker image
Contributor
|
Semgrep found 17
The method identified is susceptible to injection. The input should be validated and properly If this is a critical or high severity finding, please also link this issue in the #security channel in Slack. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Proposed changes
This is a draft PR for an ongoing S3 asset-storage spike. It is not ready to merge or presented as production-complete. The to-do list below is active work and will be worked through as this draft progresses. The immediate goal is a demonstrable filesystem/NFS-parity POC, with remaining gaps made explicit.
The spike lets dotCMS keep durable asset bytes in S3-compatible storage and restore local files on demand. Existing content URLs, filenames, permissions and content/version ownership remain in dotCMS. Local disk still provides working space and a cache for consumers that require files.
The feature is off by default, enabled with
FEATURE_FLAG_S3_ASSET_STORAGE=true(DOT_FEATURE_FLAG_S3_ASSET_STORAGE=truein Docker). Filesystem/NFS remains supported; configuringBINARY_CHAINalone does not enable the feature.Implemented so far
asset-blobs/sha256/ab/cd/ef/01/<full-64-character-hex-sha>. Existing content-owned keys contain small references. Equal bytes share a blob; deleting one owner removes its reference without deleting another owner's bytes. Legacy raw objects and the earlier one-level blob layout remain readable.asset-namespaces/<namespace>/..., while SHA blobs and byte-derived extraction remain shared. Namespaces separate application keys; they are not an IAM security boundary.Validation so far
pg_dumpcheckpoint and reindexed, then retrieved the earlier asset bytes from retained S3 data without restoring the bucket. This was not WAL/timestamp PIT recovery or automatic ownership reconciliation.The two-environment demo predates the four-level layout and latest rebase; it has not been redeployed to this branch tip. Native libvips acceptance also remains open.
To do — active work, in priority order
Deliberate limits and follow-on work
Shared blobs and extraction records are retained today. Safe reclamation, abandoned-reference reconciliation, automatic namespace decommissioning/PIT ownership reconciliation, WebDAV tombstone reclamation and shared transformation-recipe reuse are not complete. Their retention/operational limits must be documented before broader rollout; they are not prerequisites to demonstrating the scoped POC.
Reference-only/job-based push publishing is a future opportunity, not implemented by this PR. Publishing still uses bundles. Time Machine remains excluded.
Checklist and supporting documentation
See storage behavior and test instructions, the completion ledger, and the local POC fixture. The ledger contains historical checkpoints; newer results above supersede older pending/deployment notes.