Backport audit-server, SPIFFE header auth, and SQL fixes to ranger-2.10 - #1228
Open
ramackri wants to merge 67 commits into
Open
Backport audit-server, SPIFFE header auth, and SQL fixes to ranger-2.10#1228ramackri wants to merge 67 commits into
ramackri wants to merge 67 commits into
Conversation
…nsumer (#847) * RANGER-5482:Create Ranger Audit Server with SOLR and HDFS as audit consumer * RANGER-5482:Create Ranger Audit Server with SOLR and HDFS as audit consumer - fix failing testing * RANGER-5482:Create Ranger Audit Server with SOLR and HDFS as audit consumer - fix pmd issue * RANGER-5482:Create Ranger Audit Server with SOLR and HDFS as audit consumer - Fix audit commit failure propagation and recovery in the consumers * RANGER-5482:Create Ranger Audit Server with SOLR and HDFS as audit consumer - audit server partition management enhancement * RANGER-5482:Create Ranger Audit Server with SOLR and HDFS as audit consumer - Fix review comments * RANGER-5482:Create Ranger Audit Server with SOLR and HDFS as audit consumer - PojoMappingFeature for AuditEvent Object for serialization * RANGER-5482:Create Ranger Audit Server with SOLR and HDFS as audit consumer - Fix review comments set #2 * RANGER-5482:Create Ranger Audit Server with SOLR and HDFS as audit consumer - Audit Batch processing and failure reprocessing improvement * RANGER-5482:Create Ranger Audit Server with SOLR and HDFS as audit consumer - Fix duplicate dependency error in the pom for sl4j * RANGER-5482:Create Ranger Audit Server with SOLR and HDFS as audit consumer - Fix ubuntu audit ranger module war file creation failure * RANGER-5482:Create Ranger Audit Server with SOLR and HDFS as audit consumer - Fix Review comments set #3 * RANGER-5482:Create Ranger Audit Server with SOLR and HDFS as audit consumer - Fix failing test * RANGER-5482: addressed review comments/suggestions * RANGER-5482: support configuration to specify authorized users per service, instead of a global list of users * addressed review comments * cleanup in AuditConsumer implementations, RangerAuditServerDestination * added NoContentException to resolve failure in instantiating RangerJsonProvider --------- Co-authored-by: Ramesh Mani <rmani@apache.org> Co-authored-by: Madhan Neethiraj <madhan@apache.org> (cherry picked from commit 89ce14a)
#886) * RANGER-5520:Audit Server refactoring to segregate audit ingestion and dispatching functionality * RANGER-5520:Audit Server refactoring to segregate audit ingestion and dispatching functionality - copilot review comment fix * RANGER-5520:Audit Server refactoring to segregate audit ingestion and dispatching functionality - Fix review comments * RANGER-5520:Audit Server refactoring to segregate audit ingestion and dispatching functionality -Fix review comments -set #2 * RANGER-5520:Audit Server refactoring to segregate audit ingestion and dispatching functionality -Fix review comments - set #3 * addressed review suggestions * RANGER-5520:Audit Server refactoring to segregate audit ingestion and dispatching functionality - Fix review comments #4 - remove audit dispatcher registry * RANGER-5520:Audit Server refactoring to segregate audit ingestion and dispatching functionality - Fix merge conflit * RANGER-5520:Audit Server refactoring to segregate audit ingestion and dispatching functionality - fix dispatcher issue with handling failed audits * RANGER-5520:Audit Server refactoring to segregate audit ingestion and dispatching functionality - fix review comments #5 * RANGER-5520:Audit Server refactoring to segregate audit ingestion and dispatching functionality -fix review comments * RANGER-5520:Audit Server refactoring to segregate audit ingestion and dispatching functionality - Fix review comments on docker file separation for ingestor and dispatchers * RANGER-5520:Audit Server refactoring to segregate audit ingestion and dispatching functionality - Fixed review comment on docker file --------- Co-authored-by: Ramesh Mani <rmani@apache.org> Co-authored-by: Madhan Neethiraj <madhan@apache.org> (cherry picked from commit 3fd46db)
…ls sets which are already in Ranger Base image (#971) * RANGER-5613:DockerFile of Ranger Audit Server has issue duplicate tools sets which are already in Ranger Base image * RANGER-5613:DockerFile of Ranger Audit Server has issue duplicate tools sets which are already in Ranger Base image - Review Comment addressed --------- Co-authored-by: Ramesh Mani <rmani@apache.org> (cherry picked from commit e3ab2b3)
…erver (#986) * RANGER-4676, RANGER-5615: Add OpenSearch audit destination via dedicated dispatcher module * Update OpenSearchAuditDestination.java * Update AuditOpenSearchDispatcher.java * Update OpenSearchDispatcherManager.java * Update OpenSearchUtilTest.java * Update OpenSearchMgrTest.java * Update OpenSearchAccessAuditsServiceTest.java * Update OpenSearchUtil.java * Update OpenSearchAccessAuditsService.java * Update TestAuditOpenSearchDispatcher.java --------- Co-authored-by: Paras <pagnihotri@cloudera.com> Co-authored-by: Madhan Neethiraj <madhan@apache.org> (cherry picked from commit a23c30c)
…o store) via keytab-aware JAAS relogin (#1030) * RANGER-5654:Solr audit dispatcher fails to index after Kerberos TGT relogin (No key to store) with default useTicketCache=true * RANGER-5654: Restore site XML descriptions and document relogin recovery * RANGER-5654: Drop AbstractKerberosUser change; config-only fix * RANGER-5654: Keytab-aware JAAS relogin; restore useTicketCache=true Replace config-only useTicketCache=false with in-place keytab relogin in AbstractKerberosUser (agents-audit + security-admin). KerberosAction uses performRelogin() on SecurityException retry. Revert shipped and docker site XML to useTicketCache=true so the Java fix alone addresses "No key to store" at TGT renewal. Add unit test for keytab relogin without logout. * RANGER-5654: Wrap Kerberos javadoc and log lines for checkstyle Keep LineLength within 80 characters in AbstractKerberosUser, KerberosAction, and KerberosJAASConfigUser; no new checkstyle violations versus the pre-change baseline in those files. * RANGER-5654: Use single-line Kerberos relogin log statements * RANGER-5654: Address KerberosJAASConfigUser PR review feedback Cache useKeyTab in the constructor, rename helper to getBooleanOption, and use a single return per method in both agents-audit and security-admin. --------- Co-authored-by: ramk <ramk@cloudera.com> (cherry picked from commit 7017225)
…tstrap (#1138) * RANGER-5720: DB patch 078 for audit partition plan global state Add patch 078 and optimized schema updates for all DB backends: seed rangerauditserver with ROLE_ADMIN_AUDITOR and register RangerAuditPartitionPlan global state name for RANGER-5655 admin-managed partition plans. * RANGER-5720: Address Copilot review on patch 078 audit DB changes. Disable JDBC login for rangerauditserver (status=0, empty password) for SPIFFE-only Admin access; fix Oracle app_data ALTER guard; use SQL Anywhere ALTER COLUMN syntax. * RANGER-5720: Fix Oracle and SQL Server fresh-install schema for patch 078. Oracle stores empty string as NULL on NOT NULL PASSWORD; use a space placeholder for rangerauditserver. Remove stale x_policy_ref_user_FK_upd_by CHECK from SQL Server optimized schema (patch 077 regression). * RANGER-5720: Address review on MySQL LONGTEXT and Oracle CLOB upgrade. Use LONGTEXT for app_data on MySQL so the partition plan can grow beyond TEXT limits. On Oracle upgrade, migrate VARCHAR2 app_data via add/copy/drop/ rename instead of MODIFY to avoid ORA-22858. * RANGER-5720: Add x_audit_config table for audit runtime settings. Per review feedback, store ingestor URL, per-service allowed users, and topic partition count in x_audit_config (patch 078 + optimized schema on all DB backends). Seed defaults idempotently on fresh install and upgrade. * RANGER-5720: Align x_audit_config with Option A config split. Keep ingestor URL and per-service allow-list in x_audit_config; partition routing (topic, topicPartitionCount, plugins, buffer) stays in RangerAuditPartitionPlan JSON only. Drop topic-partitions from x_audit_config. * Revert "RANGER-5720: Align x_audit_config with Option A config split." This reverts commit 3b33734. * RANGER-5720: Move audit config to x_audit_config table. Store ingestor URL, topic, partition plan JSON, and per-service allow-list in x_audit_config. Revert patch 078 changes to x_ranger_global_state (no app_data widen, no RangerAuditPartitionPlan row). Keep rangerauditserver machine user setup in patch 078. * RANGER-5720: Finalize patch 078 x_audit_config and add docker DB E2E tests. Rename patch 078 to add-x_audit_config, store all audit runtime settings in x_audit_config (not x_ranger_global_state), fix Oracle upgrade SQL, drop a stale SQL Server FK CHECK from optimized schema, and add ranger-docker scripts to verify fresh install and 077→078 upgrade on all four docker DB backends. * RANGER-5720: Align SQL Server patch UK constraint naming. Use x_audit_config$x_audit_config_UK_cfg_name in patch 078 to match Ranger SQL Server convention and fresh-install schema. * RANGER-5720: Remove patch 078 docker DB test scripts. Drop db-test helpers from PR scope; patch 078 remains DB-only. --------- Co-authored-by: ramk <ramk@cloudera.com> (cherry picked from commit 2ad565f)
- Set audit-server module parent versions to 2.10.0-SNAPSHOT - Add Jersey 2 dependencyManagement for audit WAR modules - Restore 2.10 agents-common and security-admin biz code without GDS - Add AUDIT_STORE_OPENSEARCH constant for audit UI routing
…ins available (#969) * RANGER-5483:Add Audit Server as a destination for all the Ranger Plugins available * RANGER-5483:Add Audit Server as a destination for all the Ranger Plugins available - addressed review comments --------- Co-authored-by: Ramesh Mani <rmani@apache.org> (cherry picked from commit d2cd9ea)
) * RANGER-5633: Wire audit-server Kafka producer and consumer tuning Wire ingestor producer properties from site XML, apply optional topic configs at create time, default dispatchers to CooperativeStickyAssignor, and add unit tests for producer config and topic config wiring. * RANGER-5633: Wire audit-server Kafka producer and consumer tuning * RANGER-5633: Addressing Review comments --------- Co-authored-by: ramk <ramk@cloudera.com> (cherry picked from commit 4a06175)
… in Kafka and HBase plugins (#1015) * RANGER-5642, RANGER-5644: Package Jersey auditserver REST client JARs in Kafka and HBase plugins Whitelist missing Glassfish Jersey dependencies in plugin assembly descriptors so auditserver destination can POST JSON audits without MessageBodyWriter errors. * RANGER-5644: Align HBase plugin Jersey deps with HDFS/Hive for CI stability. Omit jersey-hk2 and javax.inject from the HBase plugin tarball; they crash HMaster in plugins-docker-build while entity-filtering and jersey-media-json-jackson still fix auditserver MessageBodyWriter errors. --------- Co-authored-by: ramk <ramk@cloudera.com> (cherry picked from commit 10bda4d)
…kms.xml for audit-server delivery (#1025) Add Jackson, Jersey client, and HK2 coordinates to ranger-kms-plugin-impl so the isolated plugin classloader can POST audit batches to the audit-server ingestor without LinkageError or Jersey SPI failures. https: //issues.apache.org/jira/browse/RANGER-5650 Co-authored-by: ramk <ramk@cloudera.com> (cherry picked from commit 33e7b3a)
#1020) * RANGER-5642: Exclude duplicate Jersey JARs from Kafka plugin packaging Revert the Kafka portion of #1015: the broker already ships Jersey on the application classpath; duplicate JARs in plugin-impl cause WadlAutoDiscoverable ClassCastException and audit ingestor POST failures. * RANGER-5647: Fix remaining ISO EXPIRES_ON dates in hive tag tests. PR #1018 updated slash-format tag fixture dates to 2099/12/31 but left 2026-06-15 ISO expiry_date values in test_policyengine_tag_hive.json. After 2026-06-15, TestPolicyEngine.testPolicyEngine_hiveForTag fails CI with isAllowed expected true but was false for EXPIRES_ON SELF match. * Revert "RANGER-5647: Fix remaining ISO EXPIRES_ON dates in hive tag tests." This reverts commit ef0dbdd. * RANGER-5642: Exclude duplicate Jackson JARs from Kafka plugin packaging Review feedback on #1020: broker ships Jackson 2.16.x; plugin-impl must not also whitelist jackson-annotations, jackson-core, jackson-databind, and jackson-jaxrs-json-provider at Ranger 2.17.x — same principle as the Jersey revert (use broker classpath for shared libraries). --------- Co-authored-by: ramk <ramk@cloudera.com> (cherry picked from commit 6bf1913)
) Add missing plugin-yarn template for assembly, install YARN plugin jars under share/hadoop/yarn/lib, and include slf4j-api in ranger-yarn-plugin-impl to match hdfs-agent packaging. Co-authored-by: ramk <ramk@cloudera.com> (cherry picked from commit abe6718)
…1006) * RANGER-5637: Fix Knox and Ozone plugin docker CI smoke-test failures. Use Java 17 ozone-runner (20241022-jdk17-1) and container JAVA_HOME for plugin enable so XmlConfigChanger and RangerOzoneAuthorizer load correctly. Start OM after SCM and datanode to reduce startup-order flakes. Package Jersey/HK2 and javax.inject in the Knox plugin tarball for auditserver destination after RANGER-5632, and tail gateway.log when the Knox gateway fails to start in CI. * RANGER-5637: Fix Knox and Ozone Java 17 docker smoke-test startup. Knox 2.0.0 needs JDK 17 module exports for SSL keystore generation; Ozone SCM needs -XX:-UseContainerSupport on GitHub Actions kernels with broken cgroup metrics. * RANGER-5637: Bump Knox to 2.1.0 and fix Ozone auditserver JAX-RS packaging. Knox 2.0.0 cannot generate gateway SSL certs on JDK 17 (md5WithRSAEncryption_oid removed from AlgorithmId); 2.1.0 includes KNOX-2923 JDK 17 support. Whitelist javax.ws.rs-api in Knox and Ozone plugin assemblies for auditserver REST client. * RANGER-5637: Skip redundant Ozone re-downloads in plugin archive CI step. Replace unconditional rm -rf downloads/ozone-* with extractOzoneIfNeeded(), which re-extracts only when the tree is missing, incomplete, or older than the tarball. * RANGER-5637: Stabilize Ozone extract cache with tarball stamp file. Use .ozone-extract.stamp (mtime:size) instead of directory mtime so CI skips re-extraction on warm cache hits. Document extractOzoneIfNeeded in script and workflow. * RANGER-5637: Add HK2 javax.inject to Ozone plugin and wait for SCM leader. Package org.glassfish.hk2.external:javax.inject for Jersey Singleton on the Ozone plugin classpath. Wait for SCM client port before OM plugin enable to reduce ServerNotLeaderException flakes. * RANGER-5637: Package javax and jakarta inject in Ozone plugin tarball. Declare javax.inject as a runtime dependency and include jakarta.inject in the assembly so OM auditserver JAX-RS clients resolve HK2 inject APIs. * RANGER-5637: Use multi-arch ozone-runner 20241108-jdk17-1. 20241022-jdk17-1 is amd64-only; 20241108-jdk17-1 adds arm64 for Apple Silicon without QEMU emulation. Document the choice in .env and Dockerfile.ranger-ozone. --------- Co-authored-by: ramk <ramk@cloudera.com> (cherry picked from commit 137a5dd)
…gins (#1017) * RANGER-5645: Add audit-ingestor service-user allowlist for Docker plugins Ship per-repo allowed.users and auth_to_local rules so plugins using the audit-server destination are authorized after Kerberos SPNEGO (fixes HTTP 403). Align create-ranger-services.py with policy.download.auth.users for Ozone, Atlas, Kudu, and NiFi. Add troubleshooting README for ingestor 403 errors. * RANGER-5645: Fix auth_to_local description; drop README files from PR Consolidate auth_to_local property description (JWT note + plugin rules). Revert audit-server/scripts/README.md and remove troubleshooting README. * RANGER-5645: List auth_to_local rules in Default rules provided format Match original site XML description style with one bullet per RULE line. * RANGER-5645: Limit scope to Docker services listed in Jira Remove dev_atlas, dev_kudu, and dev_nifi from ingestor allowlist, auth_to_local rules, and create-ranger-services.py (not in Docker stack). * RANGER-5645: Add dev_elasticsearch for Docker OpenSearch stack Create Policy Manager repo for the elasticsearch service type pointing at ranger-opensearch.rangernw:9200 with opensearch download auth users, and add matching ingestor allowlist plus auth_to_local rule. * RANGER-5645: Add dev_tag service and fix auth_to_local description Create dev_tag in create-ranger-services.py (matches Policy Manager). Add dev_tag ingestor allowlist (rangertagsync) and auth_to_local rule. Use CDATA so description shows <repo> instead of XML entities. * RANGER-5645: Add atlas/kudu/nifi; remove OpenSearch elasticsearch entries Match Policy Manager repos (dev_atlas, dev_kudu, dev_nifi). Drop dev_elasticsearch allowlist, auth_to_local rules, and create-service entry. * RANGER-5645: Address PR review feedback on allowlist and Docker services Remove redundant auth_to_local plugin rules covered by DEFAULT, restrict Ozone to om-only, and drop tag/atlas/kudu/nifi from create-ranger-services.py per Docker stack scope. * RANGER-5645: Fix date-bound tag policy engine tests after 2026-06-15 RESTRICTED-FINAL deny-exception uses isAccessedBefore(activation_date); fixture dates were 2026/06/15 so TestPolicyEngine_hiveForTag_filebased failed on/after that day (unrelated to ingestor allowlist changes). Use 2099/12/31 in tag test fixtures so CI stays stable. * RANGER-5645: Run TestPolicyEngine sub-tests sequentially for CI stability parallelStream() on a shared RangerPolicyEngine caused intermittent failures (e.g. hdfs_resourcespec {USER} path policy) under CI load. * RANGER-5645: Address review feedback on allowlist scope and test changes - Remove dev_tag, dev_atlas, dev_kudu, dev_nifi allowlist entries (not applicable to Docker audit-ingestor scope per review) - Revert agents-common test fixture and TestPolicyEngine changes; tag date updates belong in separate RANGER-5647 PR - Restore parallelStream() in TestPolicyEngine per review --------- Co-authored-by: ramk <ramk@cloudera.com> (cherry picked from commit 9ab0069)
Co-authored-by: ramk <ramk@cloudera.com> (cherry picked from commit ce93068)
Wave 2 cherry-picks used --theirs on .env and polluted master/JDK17 settings. Restore ranger-2.10 branch, JDK8 base images, and 2.10.0-SNAPSHOT versions; keep OPENSEARCH_VERSION for audit stack. Fix plugin-ozone parent pom.
…ouple Solr (#1057) - Extract ranger-solr from docker-compose.ranger.yml into a dedicated docker-compose.ranger-solr.yml (mirrors docker-compose.ranger-opensearch.yml) - Remove hard depends_on: ranger-solr from ranger, ranger-pdp, and ozone-om so the core, pdp, and ozone stacks start without Solr; Solr is now opt-in via -f docker-compose.ranger-solr.yml - Add OpenSearch audit setup one-liners to README, parallel to Solr/HDFS docs - Add audit_store=opensearch config block to mysql and oracle install.properties - CI (plugins-docker-build): switch from docker-compose.ranger-solr.yml to docker-compose.ranger-opensearch.yml — the plugins job has no need for Solr, and the Solr compose file's ranger-kdc dependency caused a keytab race that crashed ozone-datanode on CI runners - CI (services-docker-build): already using docker-compose.ranger-opensearch.yml (no change needed there) Co-authored-by: Paras <pagnihotri@cloudera.com> (cherry picked from commit 0249cc1)
(cherry picked from commit 22fbc81)
(cherry picked from commit b222989)
…#1139) * RANGER-5723: Plugin SPIFFE outbound auth for audit-server destination Add PluginHeaderAuthConfig, SpiffeIdentityResolver, and RangerRESTClient.setTrustedAuthHeaders(); wire SPIFFE headers into RangerAuditServerDestination when audit XML authn.header.enabled=true. * RANGER-5723: Remove unused SPIFFE prefix-discovery helpers Drop configPrefixForServiceType, resolveEnabledConfigPrefix, and RANGER_CONFIG_PREFIX; audit destination passes an explicit config prefix to buildSpiffeAuthHeaders. * RANGER-5723: Address PR review comments for SPIFFE outbound auth Remove redundant SPIFFE ID trim, add misconfiguration and REST client header tests, and document that SPIFFE header auth is additive to authn.type. * RANGER-5723: Fix Checkstyle import order in TestRangerRESTClient Add blank line between javax and java import groups required by dev-support/checkstyle.xml ImportOrder rule. * Update SpiffeIdentityResolver.java * Update SpiffeIdentityResolver.java * RANGER-5723: Fix checkstyle and address remaining PR review comments Fix SpiffeIdentityResolver checkstyle violations, rename buildSpiffeAuthHeaders to buildTrustedAuthHeaders, and add generic authn.header.headers slot-based configuration with file:/env: value resolution. * RANGER-5723: Document trusted-header config modes and value specs Expand PluginHeaderAuthConfig Javadoc for legacy SPIFFE and generic slot configuration, including file:/env:/literal value spec examples. * RANGER-5723: Use header-name-as-property for trusted auth config Address PR review: configure outbound trusted headers as authn.header.{Header-Name}=value specs (file:/env:/literal) instead of slot-based or legacy SPIFFE properties. * Update RangerAuditServerDestination.java * Update RangerAuditServerDestinationTest.java * Update TestRangerRESTClient.java * Update PluginHeaderAuthConfig.java * RANGER-5723: Drop outbound SPIFFE validation and SpiffeIdentityResolver Address PR review: trusted outbound headers pass resolved values through without SPIFFE format checks; remove SpiffeIdentityResolver and obsolete tests for the old authn.spiffe.* resolution model. * Update PluginHeaderAuthConfig.java * Update PluginHeaderAuthConfig.java * RANGER-5723: Remove unnecessary property-name sorting in header auth config. HTTP header order does not affect auth; iterate props directly per review feedback. --------- Co-authored-by: ramk <ramk@cloudera.com> Co-authored-by: Madhan Neethiraj <madhan@apache.org> (cherry picked from commit 8e7716c)
…1144) * RANGER-5736: Fix MariaDB grant failure during Ranger Admin DB setup MariaDB returns error 1133 when dba_script.py grants privileges to a host-specific user after wildcard grants with FLUSH PRIVILEGES. Detect MariaDB at runtime and adjust grant order, flush timing, and retry logic without changing behavior for Oracle MySQL or other DB flavors. * RANGER-5736: Inline MariaDB grant fix, drop extracted helpers Keep grant/flush logic inline in grant_xa_db_user; retain only detect_server_type() and user_hosts() as small MariaDB-specific helpers. * RANGER-5736: Mirror MariaDB grant fix in KMS dba_script.py Apply the same MysqlConf MariaDB detection, host order, flush timing, and grant retry logic to kms/scripts/dba_script.py for KMS Docker installs. --------- Co-authored-by: ramk <ramk@cloudera.com> (cherry picked from commit 06c44cb)
#1081) * RANGER-5693: Stop logging full JWT bearer tokens on validation failure Replace jwtToken.serialize() in the validation-failure WARN path with safeJwtLogContext() that logs only non-sensitive metadata and a SHA-256 hash for log correlation. Add null-safety for missing audience claims and unit tests for the safe logging path. * RANGER-5693: Drop tokenHash from JWT validation failure logs. Per review feedback, remove sha256Hex/tokenHash from safeJwtLogContext; jwtId is sufficient for log correlation when present. * Remove tokenHash assertion from JWT test --------- Co-authored-by: ramk <ramk@cloudera.com> (cherry picked from commit 6723329)
…ing. Revert accidental master CI migration to build-17; ranger-2.10 uses build-8 and build-11. Clear JDK 17-only JAVA_OPTS from docker .env for -8 images. Bundle Jersey/Jackson JAX-RS jars in the kafka plugin for audit-server dest.
ramackri
force-pushed
the
backport-audit-spiffe-2.10
branch
from
September 11, 2026 11:55
f642e62 to
c6b3767
Compare
added 10 commits
September 11, 2026 17:59
…ARs. Replace Glassfish Jersey 2 servlet wiring with com.sun.jersey SpringServlet so Spring beans inject on JDK 8. Drop Jersey 2 deps from ingestor and dispatcher-app; use jaxb-api instead of jakarta.xml.bind in dispatcher.
… POST. Jersey 1.x on plugin spool threads cannot reliably write List bodies, and JsonUtils date format is rejected by the audit ingestor ObjectMapper.
…ngestor. Map hive, hbase, kafka, om, and rangerkms service principals for smoke tests.
… KMS plugins. Include jackson-jaxrs and exclude Glassfish Jersey 2.x from plugin tarballs.
Use HiveConf.getConfVars for hive.exec.pre.hooks instead of removed enum.
…e audits. Fix single-node HBase startup, KMS service name and init scripts, KDC keytabs for ozone/kms/hbase, and wait for keytabs before Ozone services start.
…afka classpath. RangerRESTClient already uses com.sun.jersey types; serialization errors should not throw ClientHandlerException which breaks plugins without jersey-bundle.
…a Maven. Replace jersey-client/core with jersey-bundle in agents-common, hbase-agent, and plugin-kms so assembly filters match the dependency tree. Align kms.xml plugin-impl with Jersey 1.x packaging used by docker KMS.
…ckaging. Restore jackson-jaxrs in HBase/KMS plugin assemblies (exclude filter was dropping them), add PostgreSQL driver to Hive docker image, and wait for HS2 startup before declaring the container healthy.
Align backport Java sources with Ranger import layout and whitespace rules, convert tab-indented security-admin files to spaces, remove an unused import that failed PMD, and add dev-support formatting helpers for future use.
ramackri
force-pushed
the
backport-audit-spiffe-2.10
branch
from
September 12, 2026 03:39
20bea2e to
56abfa2
Compare
added 5 commits
September 12, 2026 09:10
…om PR. Remove format-changed-java.py and RangerCodeScheme-IntelliJ.xml added for one-time style cleanup; keep the Java source formatting changes only.
Ship jersey-bundle and javax.ws.rs-api in KMS server and plugin-impl lib sets so the Jersey 1.x REST servlet and Ranger policy client start cleanly. Fix Knox image directory permissions, JAVA_HOME, and preserve env for gateway startup; align Solr 9 entrypoint and Kerberos settings with master.
knoxcli create-master runs as root and leaves /opt/knox/data/security unreadable by the knox user; chown data/logs/pids after expect so gateway startup succeeds.
knox-jdk17-env.sh --add-exports options are rejected by Java 8 and prevent LDAP/gateway startup on the ranger-2.10 base image.
Package Jersey 1.x correctly for KMS, Knox, and Solr plugins, add audit-core to Knox/Solr assemblies, fix OpenSearch dispatcher duplicate Spring jars, and correct KMS auth_to_local rules so keyadmin mapping works in docker.
ramackri
force-pushed
the
backport-audit-spiffe-2.10
branch
from
September 12, 2026 06:24
caf47f3 to
cd9daaa
Compare
added 11 commits
September 12, 2026 12:33
RangerAdminJersey2RESTClient requires JAX-RS 2.0 Response APIs at compile time; plugin runtime packaging still uses Jersey 1.x via assembly excludes.
com.sun.jersey jersey-client pulled JAX-RS 1.x jsr311-api, which shadowed javax.ws.rs 2.x Response APIs required by RangerAdminJersey2RESTClient.
Reorder provided-scope dependencies so sortpom:verify passes in CI.
Install the Knox plugin jar from build-8 before the JDK 11 reactor run, start Solr in services/plugins docker compose (ranger-solr was checked but never brought up), and drop references to the missing OpenSearch compose file on the 2.10 branch.
Maven install:install-file does not expand globs; resolve the downloaded artifact path with find before installing to the local repository.
Use docker-compose.ranger-audit-service.yml with the audit-store-opensearch profile (matching RANGER-5680), copy audit ingestor/dispatcher tarballs, and check ranger-opensearch plus the audit pipeline containers instead of Solr.
Remove external: true on rangernw so docker compose can create the network when bringing up admin, audit-service, and PDP together.
Stage docker compose to wait for Ranger Admin before starting the audit pipeline, give Kafka more healthcheck time for policy cache sync, and install KMS from build-8 on the JDK 11 job (RangerMasterKeyTest fails on 11).
Use ranger-db (the compose service name) instead of ranger-postgres when starting the core stack; ranger-postgres is only the container name.
Point PDP at the audit ingestor instead of Solr, restore ranger-pdp Kerberos keytabs dropped during the audit backport, and fix Hive schematool ordering and Kerberos HDFS access for plugins-docker CI.
Backport master ranger-hive.sh log4j2 and service readiness checks, create audit spool directories before enabling the plugin, and allow more time for HiveServer2 to start before the container health check.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Ranger Audit Server (foundation → plugin wiring → docker)
x_audit_configtable and audit server bootstrapPlugin packaging & docker CI (audit-server delivery)
kms.xmlSPIFFE / header-based authentication
Extends RANGER-5499 already on 2.10.
SQL / admin fixes
Backport-specific fixes (2.10 / JDK 8)
These commits adapt master changes for
ranger-2.10and are not direct cherry-picks:hive.storage-api.versionin HDFS dispatcher; Kylin RoaringBitmap → Maven Central0.7.36; RAT license check fix; remove sortpom backup filescom.sun.jerseySpringServlet (not Glassfish Jersey 2); plugin REST clients stay on Jersey 1.x with SPIFFE outbound headers only; pre-serialize audit batches for audit-server POST; avoidClientHandlerExceptionon Kafka classpath; pulljersey-bundleinto plugin assemblies via Maven; package Jackson/JAX-RS in HBase/KMS/Kafka pluginsbuild-8/build-11(revert accidental masterbuild-17); JDK8-compatible SPIFFE and OpenSearch unit tests (Map.of/List.of/Files.writeStringreplacements).envafter Wave 2 conflicts; Kerberosauth_to_localrules for audit ingestor; harden HBase/KMS/Ozone smoke (KDC keytabs, single-node HBase, KMS init); Hive 4.0ConfVarslookup fix; PostgreSQL driver in Hive docker image; fix jackson-jaxrs exclude filters in HBase/KMS assemblies--theirson shared admin/agents-common files; security-admin build fixes without GDSExplicitly excluded from this backport
Ranger Audit Server — what we tested
Audit pipeline topology (OpenSearch default, RANGER-5680)
Docker CI and local smoke use the audit-service compose stack with
--profile audit-store-opensearch(not standalone Solr):Audit server components verified
services-docker-build+plugins-docker-buildcontainer check:7093; indexes events into OpenSearchranger-opensearch):9200ranger-pdpAudit destinations wired in docker
ranger.authz.audit.destination.auditserver=true→http://ranger-audit-ingestor.rangernw:7081(Solr disabled)SolrAuditDestinationClassNotFoundExceptionXAAUDIT.AUDITSERVER.ENABLE=trueinranger-hive-plugin-install.propertiesranger-audit-dest-auditserverJAR in plugin-impl assembliesEnd-to-end audit indexing (local smoke)
Verified audit events flow through the full pipeline into OpenSearch indices:
dev_kafkaindex (+140 during smoke)dev_hdfsindex (+13)dev_hiveaudit count increased after HS2 + plugin enable:7093after removing duplicatespring-webfrom dispatcher libNot tested on this branch
x_audit_config) on fresh DB installs outside docker PostgresTest plan
Build & unit compile (local, JDK 8)
mvn install -pl audit-server/audit-common,audit-server/audit-ingestor,audit-server/audit-dispatcher,agents-audit/dest-auditserver,agents-audit/dest-os,common-utils -am -DskipTests -Drat.skip=truemvn compile -pl security-admin -am -DskipTests -Drat.skip=truemvn verify -pl audit-server,agents-audit/dest-auditserver,agents-audit/dest-os,agents-common -DskipTests -Drat.skip=true(checkstyle + PMD on backport modules)mvn pmd:check -pl security-admin,agents-common,pdp,embeddedwebserver,hive-agent,kms -DskipTests -Drat.skip=true(PMD clean after unused-import fix)SpiffeIdUtilTest,OpenSearchUtilTest, audit-server dispatcher tests)Packaging verification (local docker build)
docker compose run ranger-buildwithmvn -pl distro -am(not distro-only — avoids empty ~19 KB plugin tarballs)jersey-bundle-1.19.4.jar(Kafka, HBase, KMS, Knox) after POM/assembly alignmentjackson-jaxrs-*JARs after removing conflicting assembly excludesspring-webduplicate-class conflictDocker smoke — audit server + plugins (local)
9200)jackson-jaxrspackaging fixCI (GitHub Actions) — run 34700451794
build-8— passbuild-11— pass (Knox/KMS jars installed from build-8; modules excluded from JDK 11 compile)services-docker-build— pass (Admin, PDP, usersync, tagsync, KMS, Kafka, audit-ingestor, audit-dispatcher-opensearch, ranger-opensearch)plugins-docker-build— pass (Hadoop, HBase, Hive, Knox, Ozone + audit pipeline containers)docker-compose.ranger-audit-service.yml+--profile audit-store-opensearchAdmin / DB / auth — pending
x_audit_config) on a test Postgres/MySQL/Oracle instance