feat(store): implement pluggable cloud storage for HStore

[vaijosh]

Purpose

• Closes: [Feature] Implement Cloud Storage Support for HugeGraph #3079

This PR introduces a cloud storage architecture for HugeGraph Store (HStore). In cloud-native environments, storage nodes are ephemeral, making reliance on local disk storage a single point of failure. This implementation decouples the storage layer from local disk dependencies, allowing SST files to be offloaded to durable, scalable cloud storage providers.

Key Changes

Pluggable Architecture

• Created the CloudStorageProvider SPI interface to enable extensible storage backends.
• Added CloudStorageProviderFactory for seamless provider discovery and lifecycle management.

S3 Provider Implementation

• Introduced the hg-store-cloud-s3 module, leveraging AWS SDK v2 for production-ready S3/S3 compatible storage interaction.

Lifecycle & Event Integration

• Integrated CloudStorageEventListener with RocksDBFactory. This hooks into critical SST lifecycle events (onTableFileCreated, onTableFileDeleted) to ensure synchronized state between local RocksDB and cloud storage.
• Implemented startup hydration (onDBCreated) to backfill pre-existing files and read-miss on-demand hydration to ensure data availability.

Infrastructure & Testing

• Added a comprehensive local development environment via docker/cloud-storage/ using MinIO.
• Included an integration test script (test-graph-queries-and-sst.sh) to verify end-to-end data durability and query consistency.

Implementation Highlights

• SPI Integration: Standardized service discovery via META-INF/services.
• RocksDB Hooking: Registered a singleton RocksdbEventListener within RocksDBFactory to intercept file operations without modifying core RocksDB logic.
• Read-Miss Protection: Added a guard window mechanism in onReadMiss to prevent redundant hydration requests during high-concurrency read scenarios.

Verifying These Changes

• Unit Tests: Coverage for config parsing (CloudStorageConfigTest), factory registration, and event listener logic.
• Integration Tests: Verified using the new docker-compose setup with MinIO.
• Configuration: Verified application.yml bindings for credentials, bucket management, and sync intervals.

Impact

• Dependencies: Added AWS SDK v2; updated LICENSE and NOTICE accordingly.
• Configurations: Added cloud.storage namespace to application.yml (disabled by default).
• Public API: No breaking changes to existing public-facing client APIs.

Documentation

• Architecture and configuration details are documented in hugegraph-store/docs/pluggable-cloud-storage-architecture.md.

[codecov]

Codecov Report

❌ Patch coverage is 94.79167% with 5 lines in your changes missing coverage. Please review.
✅ Project coverage is 30.60%. Comparing base (e960cc5) to head (a560ad4).

Files with missing lines	Patch %	Lines
.../hugegraph/store/business/BusinessHandlerImpl.java	0.00%	4 Missing ⚠️
...ache/hugegraph/store/cloud/CloudStorageConfig.java	93.33%	0 Missing and 1 partial ⚠️

❗ There is a different number of reports uploaded between BASE (e960cc5) and HEAD (a560ad4). Click for more details.

HEAD has 3 uploads less than BASE

Flag	BASE (e960cc5)	HEAD (a560ad4)
	4	1

Additional details and impacted files

@@              Coverage Diff              @@
##             master    #3081       +/-   ##
=============================================
- Coverage     42.32%   30.60%   -11.72%     
+ Complexity      460      411       -49     
=============================================
  Files           819      812        -7     
  Lines         70745    69690     -1055     
  Branches       9366     9177      -189     
=============================================
- Hits          29944    21331     -8613     
- Misses        37705    45935     +8230     
+ Partials       3096     2424      -672     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[VGalaxies]

Review summary

• Blocking: yes
• Summary: The change cannot currently guarantee recoverable RocksDB state, may silently disable cloud storage, and contains S3 retry and deployment defects.
• Evidence:
  • Static analysis of git diff origin/master...HEAD
  • shell syntax and POM XML checks passed
  • git diff --check reported whitespace-only errors

[VGalaxies]

🚨 Review summary

Important

The cloud recovery path has blocking consistency, configuration, deletion-safety, release-metadata, and verification gaps.

📊 Risk dashboard

Signal	Result
🚦 Review gate	Blocked
🔎 Actionable findings	9 (8 High, 1 Medium)
🧪 Verification coverage	5 checks
⚠️ Confirmed / limited checks	2 / 1

🔬 Coverage details

⚠️ S3 failure semantics — Confirmed

Trace: S3CloudStorageProvider upload, download, delete, and exception-classification paths, CloudStorageEventListener purge and hydration flows, AWS SDK 2.33.8 retry-code and file-transformer behavior

Conclusion: Confirmed ignored per-object deletion errors, retryable error-code misclassification, and non-atomic download recovery risks; lower-impact lifecycle leads were dropped.

⚠️ Release and E2E integrity — Confirmed

Trace: install-dist release LICENSE and known-dependencies inventory, cloud-storage Docker artifact build and recovery workflow, Maven provider packaging and test activation

Conclusion: Confirmed the release dependency inventory mismatch and E2E false-green paths caused by suppressed build and wipe failures.

🟡 Focused unit suites — Limited

Trace: hg-store-common cloud configuration, provider, and factory tests, hg-store-cloud-s3 exception-classification tests, hg-store-node listener, retry, metrics, tracker, configuration, and callback tests, hg-store-core BusinessHandlerImplTest

Conclusion: Surefire reports recorded 184 tests with 0 failures, 0 errors, and 0 skipped. The suites do not cover concurrent metadata publication, comma-separated data roots, or partial S3 batch deletion.

✅ Maven reactor — Clear

Trace: Full 44-project reactor with the cloud-s3 profile, hugegraph-store module ordering and dependency graph

Conclusion: Maven validate completed successfully for all 44 reactor projects; the suspected duplicate cloud-s3 module activation did not reproduce.

✅ Shell syntax — Clear

Trace: pd-entrypoint.sh, store-entrypoint.sh, test-graph-queries-and-sst.sh

Conclusion: bash -n returned successfully for all three changed shell entrypoints and workflows.

Warning

Verification limits

• No live Docker and MinIO recovery drill was run, so cluster recovery and purge behavior were not exercised end to end.
• The test environment used Java 17; the legacy JaCoCo agent emitted unsupported class-version instrumentation errors, so coverage instrumentation was not verified even though test assertions passed.
• S3SingleLargeFileE2ETest was intentionally not run because its default path generates a 20 GiB file.
• The independent recovery-consistency run timed out after 900 seconds and produced no usable result.

---

_{🤖 Codex review · GPT-5.6 Sol · effort: xhigh}

[vaijosh]

🚨 Review summary

Important

The cloud recovery path has blocking consistency, configuration, deletion-safety, release-metadata, and verification gaps.

📊 Risk dashboard

Signal Result
🚦 Review gate Blocked
🔎 Actionable findings 9 (8 High, 1 Medium)
🧪 Verification coverage 5 checks
⚠️ Confirmed / limited checks 2 / 1

🔬 Coverage details

⚠️ S3 failure semantics — Confirmed
Trace: S3CloudStorageProvider upload, download, delete, and exception-classification paths, CloudStorageEventListener purge and hydration flows, AWS SDK 2.33.8 retry-code and file-transformer behavior

Conclusion: Confirmed ignored per-object deletion errors, retryable error-code misclassification, and non-atomic download recovery risks; lower-impact lifecycle leads were dropped.

⚠️ Release and E2E integrity — Confirmed
Trace: install-dist release LICENSE and known-dependencies inventory, cloud-storage Docker artifact build and recovery workflow, Maven provider packaging and test activation

Conclusion: Confirmed the release dependency inventory mismatch and E2E false-green paths caused by suppressed build and wipe failures.

🟡 Focused unit suites — Limited
Trace: hg-store-common cloud configuration, provider, and factory tests, hg-store-cloud-s3 exception-classification tests, hg-store-node listener, retry, metrics, tracker, configuration, and callback tests, hg-store-core BusinessHandlerImplTest

Conclusion: Surefire reports recorded 184 tests with 0 failures, 0 errors, and 0 skipped. The suites do not cover concurrent metadata publication, comma-separated data roots, or partial S3 batch deletion.

✅ Maven reactor — Clear
Trace: Full 44-project reactor with the cloud-s3 profile, hugegraph-store module ordering and dependency graph

Conclusion: Maven validate completed successfully for all 44 reactor projects; the suspected duplicate cloud-s3 module activation did not reproduce.

✅ Shell syntax — Clear
Trace: pd-entrypoint.sh, store-entrypoint.sh, test-graph-queries-and-sst.sh

Conclusion: bash -n returned successfully for all three changed shell entrypoints and workflows.

Warning

Verification limits

• No live Docker and MinIO recovery drill was run, so cluster recovery and purge behavior were not exercised end to end.
• The test environment used Java 17; the legacy JaCoCo agent emitted unsupported class-version instrumentation errors, so coverage instrumentation was not verified even though test assertions passed.
• S3SingleLargeFileE2ETest was intentionally not run because its default path generates a 20 GiB file.
• The independent recovery-consistency run timed out after 900 seconds and produced no usable result.

🤖 Codex review · GPT-5.6 Sol · effort: xhigh

Hi @VGalaxies.
Thanks for reviewing the changes. I have addressed all the review comments. Can you please review it again?

[vaijosh]

@VGalaxies
I am not sure why code coverage check is failing.
We have
The Diff has 94.79% coverage (agains target 42.32%) [codecov/patch](https://github.com/apache/hugegraph/pull/3081/checks?check_run_id=89150435397)Successful in 1s — 94.79% of diff hit (target 42.32%)

Not sure why the coverage is suddenly dropped at project level to 30.60% (-11.72%) compared to e960cc5 codecov/projectFailing after 1s — 30.60% (-11.72%) compared to e960cc5