Skip to content

feat(storage): add multi-realm RGW support and phase-resume fixes - #22

Open
htohru-ibm wants to merge 1 commit into
mainfrom
feat/setup-storage-multi-realm-rgw
Open

feat(storage): add multi-realm RGW support and phase-resume fixes#22
htohru-ibm wants to merge 1 commit into
mainfrom
feat/setup-storage-multi-realm-rgw

Conversation

@htohru-ibm

Copy link
Copy Markdown
Member

Summary

Enhancements to scripts/setup-storage.sh to support multi-realm Ceph
clusters and fix several phase-resume failures.

Changes

Multi-realm RGW support (--rgw-realm)

  • Add --rgw-realm REALM option for native-mode clusters that have named
    realms (i.e. not the built-in default realm).
  • Rewrite detect_rgw_zone() for reliable zone name resolution:
    • When --rgw-realm is provided, use radosgw-admin zone get --rgw-realm=<realm> to obtain the zone name. zone list's
      default_info field contains a UUID, not the zone name, so it cannot
      be used for this purpose.
    • Redirect all debug/warning output to stderr so that the zone name
      captured via $() is not polluted.
    • In the fallback path, extract the zone name from the zones array
      instead of default_info.

Resolve RGW host short name to IP address

  • ceph orch ps returns a short hostname that is typically not resolvable
    from an external tenant cluster.
  • After obtaining RGW_HOST from the orchestrator, resolve it to the
    node's public IP address using ceph orch host ls -f json + jq.
    If resolution fails, the short hostname is kept as-is.

Auto-detect RGW protocol (http/https)

  • Inspect ceph config dump for ssl_port / ssl_certificate in the
    rgw_frontends config and set RGW_PROTOCOL=https automatically when
    SSL is detected.

Script name correction

  • Replace all references to ceph-external-cluster-details-exporter.py
    with create-external-cluster-resources.py, which is the current name
    used by Rook/ODF.

Phase-resume credential restoration (phases 6, 7, 8)

  • When restarting from a mid-run phase with --phase N, RGW_ACCESS_KEY,
    RGW_SECRET_KEY, and MAIN_RGW_ENDPOINT could be unset because phase 1
    was skipped. These are now restored from the credentials file written by
    phase 5. If MAIN_RGW_ENDPOINT is missing from the file, it is
    re-detected via ceph orch ps --daemon-type rgw.

Other

  • Add MAIN_RGW_ENDPOINT_OVERRIDE env var to allow explicit endpoint
    override without modifying the script.
  • Add --connect-timeout 10 --max-time 20 to the backing-bucket curl
    call for reliable timeout behaviour on slow or unresponsive endpoints.
  • Delete the state file in cleanup_artifacts() so the script can be
    re-run from scratch after --delete.

Testing

Tested against a native Ceph cluster with:

  • Single-realm (default) setup — existing behaviour unchanged
  • Multi-realm setup (--rgw-realm realm-ocp1) — correct zone resolution
  • Phase 6/7/8 resume after a mid-run failure — credentials restored correctly

Key changes
-----------
- Add --rgw-realm option for multi-realm Ceph clusters (native mode).
  Required when the cluster has named realms instead of 'default'.

- Rewrite detect_rgw_zone() for reliable zone resolution:
  - Use 'radosgw-admin zone get --rgw-realm=<realm>' when --rgw-realm
    is given (default_info in zone list contains a UUID, not the name).
  - Redirect debug/warning to stderr so the zone name captured via $()
    is not polluted.
  - Extract zone name from the 'zones' array in the fallback path.

- Resolve RGW_HOST from short hostname to IP address:
  'ceph orch ps' returns a short hostname that is typically not resolvable
  from an external tenant cluster.  Use 'ceph orch host ls -f json' + jq
  to map the hostname to its public IP address automatically.
  If resolution fails the short hostname is kept as-is.

- Auto-detect RGW protocol (http/https) from 'ceph config dump' by
  checking the rgw_frontends config for ssl_port / ssl_certificate.

- Rename ceph-external-cluster-details-exporter.py references to
  create-external-cluster-resources.py (current Rook/ODF script name).

- Restore RGW credentials and MAIN_RGW_ENDPOINT when resuming from
  phase 6, 7, or 8 (previously caused failures on --phase N reruns).
  Falls back to 'ceph orch ps --daemon-type rgw' detection if the
  credentials file does not yet contain the RGW_ENDPOINT entry.

- Add MAIN_RGW_ENDPOINT_OVERRIDE env var to allow explicit endpoint
  override without modifying the script.

- Add --connect-timeout/--max-time to backing-bucket curl call for
  reliable timeout behaviour on slow/unresponsive endpoints.

- Delete state file in cleanup_artifacts() so the script can be
  re-run from scratch after --delete.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant