RustFS has landed On-Demand Migration (ODM): a bucket can name an external S3-compatible source bucket, and a GET for a key that does not exist locally is fetched from that source, streamed to the client, and stored locally in the same pass. It is the RustFS counterpart of Cloudflare R2 Sippy, Tigris shadow buckets, and Alibaba Cloud OSS / Tencent COS mirror-back-to-origin, and it is the answer to the long-standing request for inbound/pull replication from a generic S3 source.
This issue asks for a bilingual (English + Chinese) documentation page for it.
Requested page
- Path:
administration/data/bucket/on-demand-migration (both locales), next to the other bucket-level data-management pages.
- Source content:
docs/operations/on-demand-migration.md in the rustfs/rustfs repository. That guide is the source of truth; the site page should be a user-facing rendering of it, not a fork of its content.
What the page should cover
- What ODM is, and how it differs from bucket replication (outbound push), site replication, tiering/ILM, and the replication read-proxy. The positioning table in the guide can be reused directly.
- Enabling a source: the
RUSTFS_ON_DEMAND_MIGRATION_ENABLED module switch, then the admin API — PUT / PUT ?dry-run=true / GET / DELETE /rustfs/admin/v3/on-demand-migration/{bucket} and GET .../{bucket}/status, with SigV4-signed examples.
- The complete configuration reference (
source, filter, policy) with defaults and bounds.
- Observable semantics: GET/HEAD/LIST/PUT/DELETE, Range GET,
versionId, conditional headers, delete markers, SSE-C sources, events, replication, quota and Object Lock interactions, and the x-rustfs-on-demand-migration: source response marker.
- Provider presets (
s3, aws, minio, rustfs, r2, gcs) and the least-privilege source policy: s3:ListBucket, s3:GetObject, plus s3:GetObjectTagging only when tag copying is enabled.
- Integrity and ETag rules, the copied-metadata allowlist, and the provenance metadata written on every pulled object.
- Protections: circuit breaker, negative cache, singleflight, concurrency limit, bounded pull queue, bandwidth limit, retries, anti-loop markers.
- Error codes (424
SourceUnavailable and its failure classes, the 404 degradation policy, the admin-side codes), observability (the rustfs_on_demand_migration_* metrics, PromQL examples, the status fields), troubleshooting, known limitations, and the security notes — in particular that source credentials live in bucket metadata unencrypted and should be read-only and scoped.
Notes for the writer
- Never show a real access key or secret in examples; the upstream guide uses
AKIA… and REDACTED placeholders.
- The background backfill job is a separate slice; the guide marks it as such. Please keep the same caveat until it ships, rather than documenting it as available.
- Two status fields are intentionally
null in the current build (breaker.opened_at, served_by_source_ratio); do not describe them as broken.
References
RustFS has landed On-Demand Migration (ODM): a bucket can name an external S3-compatible source bucket, and a GET for a key that does not exist locally is fetched from that source, streamed to the client, and stored locally in the same pass. It is the RustFS counterpart of Cloudflare R2 Sippy, Tigris shadow buckets, and Alibaba Cloud OSS / Tencent COS mirror-back-to-origin, and it is the answer to the long-standing request for inbound/pull replication from a generic S3 source.
This issue asks for a bilingual (English + Chinese) documentation page for it.
Requested page
administration/data/bucket/on-demand-migration(both locales), next to the other bucket-level data-management pages.docs/operations/on-demand-migration.mdin therustfs/rustfsrepository. That guide is the source of truth; the site page should be a user-facing rendering of it, not a fork of its content.What the page should cover
RUSTFS_ON_DEMAND_MIGRATION_ENABLEDmodule switch, then the admin API —PUT/PUT ?dry-run=true/GET/DELETE /rustfs/admin/v3/on-demand-migration/{bucket}andGET .../{bucket}/status, with SigV4-signed examples.source,filter,policy) with defaults and bounds.versionId, conditional headers, delete markers, SSE-C sources, events, replication, quota and Object Lock interactions, and thex-rustfs-on-demand-migration: sourceresponse marker.s3,aws,minio,rustfs,r2,gcs) and the least-privilege source policy:s3:ListBucket,s3:GetObject, pluss3:GetObjectTaggingonly when tag copying is enabled.SourceUnavailableand its failure classes, the 404 degradation policy, the admin-side codes), observability (therustfs_on_demand_migration_*metrics, PromQL examples, the status fields), troubleshooting, known limitations, and the security notes — in particular that source credentials live in bucket metadata unencrypted and should be read-only and scoped.Notes for the writer
AKIA…andREDACTEDplaceholders.nullin the current build (breaker.opened_at,served_by_source_ratio); do not describe them as broken.References