HDDS-11454. Ranger integration for Docker Compose environment #8575

adoroszlai · 2025-06-06T11:50:57Z

What changes were proposed in this pull request?

Add Docker Compose add-on in ozonesecure-ha for running tests with Apache Ranger.

The test script currently only starts the environment, but does not execute any Robot tests yet. I think that will require setting up policies for Ozone in Ranger.

The environment uses my own Docker image for Ranger, because official Apache Ranger image is available only for arm64 (see #8037). If they are adopted by Apache Ranger, we can switch to official images.

https://issues.apache.org/jira/browse/HDDS-11454

How was this patch tested?

cd hadoop-ozone/dist/target/ozone-2.1.0-SNAPSHOT/compose/ozonesecure-ha
KEEP_RUNNING=true ./test-ranger.sh

Opened Ranger Admin, verified it works:

open http://localhost:6080

Verified OM started without problems.

CI:

Executing test ozonesecure-ha/test-ranger.sh
...
Port 6080 is available on ranger

https://github.com/adoroszlai/ozone/actions/runs/15488993530/job/43610706919

adoroszlai · 2025-06-06T13:34:50Z

@kumaab please take a look, too

kumaab · 2025-06-06T21:00:13Z

hadoop-ozone/dist/src/main/compose/common/ranger.yaml

+#
+# This requires Apache Ranger source to be available in $RANGER_SOURCE_DIR.
+
+services:


I'm assuming ranger-solr and ranger-zk are not required for integration tests, Solr is the default audit destination for Ranger. Functionally the tests would pass but with no audit destination configured, audit related errors will be seen in ranger logs.

kumaab · 2025-06-06T21:22:26Z

Hi @adoroszlai,

I reviewed the patch, it looks good to me, I'll follow-up on revising the patch for #8037 using this patch.
Multi-arch support for Ranger base image is currently a work in progress RANGER-5158, it would be helpful to have your changes be part of the Ranger codebase.

adoroszlai · 2025-06-13T03:14:21Z

@smengcl can you please also take a look?

dombizita

Thanks for working on this @adoroszlai! Also for improving the patch, I tested it locally on my Mac and it worked, I was able to login to the Ranger UI, also Ozone was up.

adoroszlai · 2025-06-16T03:11:37Z

Thanks @dombizita, @kumaab for the review.

…#8575)

yazynin · 2025-06-16T13:43:51Z

Thanks for working on this @adoroszlai! Also for improving the patch, I tested it locally on my Mac and it worked, I was able to login to the Ranger UI, also Ozone was up.

@dombizita Hello, you can confirm, if you go to localhost:6080/index.html#/service/201/edit/11. and click test connection it worked?
more details https://issues.apache.org/jira/browse/RANGER-4411, thanks!

…239-container-reconciliation Commits: 62 da53b5b HDDS-13299. Fix failures related to delete (apache#8665) 8c1b439 HDDS-13296. Integration check always passes due to missing output (apache#8662) 7329859 HDDS-13023. Container checksum is missing after container import (apache#8459) a0af93e HDDS-13292. Change `<? extends KeyValue>` to `<KeyValue>` in test (apache#8657) f3050cf HDDS-13276. Use KEY_ONLY/VALUE_ONLY iterator in SCM/Datanode. (apache#8638) e9c0a45 HDDS-13262. Simplify key name validation (apache#8619) f713e57 HDDS-12482. Avoid using CommonConfigurationKeys (apache#8647) b574709 HDDS-12924. datanode used space calculation optimization (apache#8365) de683aa HDDS-13263. Refactor DB Checkpoint Utilities. (apache#8620) 97262aa HDDS-13256. Updated OM Snapshot Grafana Dashboard to reflect metric updates from HDDS-13181. (apache#8639) 9d2b415 HDDS-13234. Expired secret key can abort leader OM startup. (apache#8601) d9049a2 HDDS-13220. Change Recon 'Negative usedBytes' message loglevel to DEBUG (apache#8648) 6df3077 HDDS-9223. Use protobuf for SnapshotDiffJobCodec (apache#8503) a7fc290 HDDS-13236. Change Table methods not to throw IOException. (apache#8645) 9958f5b HDDS-13287. Upgrade commons-beanutils to 1.11.0 due to CVE-2025-48734 (apache#8646) 48aefea HDDS-13277. [Docs] Native C/C++ Ozone clients (apache#8630) 052d912 HDDS-13037. Let container create command support STANDALONE , RATIS and EC containers (apache#8559) 90ed60b HDDS-13279. Skip verifying Apache Ranger binaries in CI (apache#8633) 9bc53b2 HDDS-11513. All deletion configurations should be configurable without restart (apache#8003) ac511ac HDDS-13259. Deletion Progress - Grafana Dashboard (apache#8617) 3370f42 HDDS-13246. Change `<? extend KeyValue>` to `<KeyValue>` in hadoop-hdds (apache#8631) 7af8c44 HDDS-11454. Ranger integration for Docker Compose environment (apache#8575) 5a3e4e7 HDDS-13273. Bump awssdk to 2.31.63 (apache#8626) 77138b8 HDDS-13254. Change table iterator to optionally read key or value. (apache#8621) ce288b6 HDDS-13265. Simplify the page Access Ozone using HTTPFS REST API (apache#8629) 36fe888 HDDS-13275. Improve CheckNative implementation (apache#8628) d38484e HDDS-13274. Bump sqlite-jdbc to 3.50.1.0 (apache#8627) 3f3ec43 HDDS-13266. `ozone debug checknative` to show OpenSSL lib (apache#8623) 8983a63 HDDS-13272. Bump junit to 5.13.1 (apache#8625) a927113 HDDS-13271. [Docs] Minor text updates, reference links. (apache#8624) 7e77058 HDDS-13112. [Docs] OM Bootstrap can also happen when follower falls behind too much. (apache#8600) fd13300 HDDS-10775. Support bucket ownership verification (apache#8558) 3ecf345 HDDS-13207. [Docs] Third party systems compatible with Ozone S3. (apache#8584) ad5a507 HDDS-13035. SnapshotDeletingService should hold write locks while purging deleted snapshots (apache#8554) 38a9186 HDDS-12637. Increase max buffer size for tar entry read/write (apache#8618) f31c264 HDDS-13045. Implement Immediate Triggering of Heartbeat when Volume Full (apache#8590) 0701d6a HDDS-13248. Remove `ozone debug replicas verify` option --output-dir (apache#8612) ca1afe8 HDDS-13257. Remove separate split for shell integration tests (apache#8616) 5d6fe94 HDDS-13216. Standardize Container[Replica]NotFoundException messages (apache#8599) 1e47217 HDDS-13168. Fix error response format in CheckUploadContentTypeFilter (apache#8614) 6d4d423 HDDS-13181. Added metrics for internal Snapshot Operations. (apache#8606) 4a461b2 HDDS-10490. Intermittent NPE in TestSnapshotDiffManager#testLoadJobsOnStartUp (apache#8596) bf29f7f HDDS-13235. The equals/hashCode methods in anonymous KeyValue classes may not work. (apache#8607) 6ff3ad6 HDDS-12873. Improve ContainerData statistics synchronization. (apache#8305) 09d3b27 HDDS-13244. TestSnapshotDeletingServiceIntegrationTest should close snapshots after deleting them (apache#8611) 931bc2d HDDS-13243. copy-rename-maven-plugin version is missing (apache#8605) 3b5985c HDDS-13244. Disable TestSnapshotDeletingServiceIntegrationTest 6bf009c HDDS-12927. metrics and log to indicate datanode crossing disk limits (apache#8573) 752da2b HDDS-12760. Intermittent Timeout in testImportedContainerIsClosed (apache#8349) 8c32363 HDDS-13050. Update StartFromDockerHub.md. (apache#8586) ba1887c HDDS-13241. Fix some potential resource leaks (apache#8602) bbaf71e HDDS-13130. Rename all instances of Disk Usage to Namespace usage (apache#8571) 0628386 HDDS-13142. Correct SCMPerformanceMetrics for delete operation. (apache#8592) 516bc96 HDDS-13148. [Docs] Update Transparent Data Encryption doc. (apache#8530) 5787135 HDDS-13229. [Doc] Fix incorrect CLI argument order in OM upgrade docs (apache#8598) ba95074 HDDS-13107. Support limiting output of `ozone admin datanode list` (apache#8595) e7f5544 HDDS-13171. Replace pipelineID if nodes are changed (apache#8562) 3c9d4d8 HDDS-13103. Correct transaction metrics in SCMBlockDeletingService. (apache#8516) f62eb8a HDDS-13160. Remove SnapshotDirectoryCleaningService and refactor AbstractDeletingService (apache#8547) b46e6b2 HDDS-13150. Fixed SnapshotLimitCheck when failures occur. (apache#8532) 203c1d3 HDDS-13206. Update documentation for Apache Ranger (apache#8583) 2072ef0 HDDS-13214. populate-cache fails due to unused dependency (apache#8594) Conflicts: hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/impl/ContainerData.java hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/keyvalue/KeyValueContainer.java hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/keyvalue/helpers/KeyValueContainerUtil.java hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/keyvalue/statemachine/background/BlockDeletingTask.java

dombizita · 2025-06-27T09:22:00Z

@yazynin sorry for coming back late, I forgot to check this. I just did now and it doesn't work:

Connection Failed
Unable to retrieve any files using given parameters, You can still save the repository and start creating policies, but you would not be able to use autocomplete for resource names. Check ranger_admin.log for more info.

HDDS-11454. Ranger integration for Docker dev environment

5d53489

adoroszlai self-assigned this Jun 6, 2025

adoroszlai added the test label Jun 6, 2025

adoroszlai requested a review from smengcl June 6, 2025 13:22

kumaab reviewed Jun 6, 2025

View reviewed changes

kumaab added a commit to kumaab/ozone that referenced this pull request Jun 8, 2025

improvise using PR apache#8575

b63227a

adoroszlai requested a review from dombizita June 10, 2025 08:45

adoroszlai added 3 commits June 10, 2025 17:35

replace sed with perl for portability

4dee960

make scripts executable

6db2e0e

Merge remote-tracking branch 'origin/master' into HDDS-11454

3eb78bb

dombizita approved these changes Jun 15, 2025

View reviewed changes

kumaab approved these changes Jun 16, 2025

View reviewed changes

adoroszlai merged commit 7af8c44 into apache:master Jun 16, 2025
27 checks passed

adoroszlai deleted the HDDS-11454 branch June 16, 2025 03:11

sadanand48 pushed a commit to sadanand48/hadoop-ozone that referenced this pull request Jun 16, 2025

HDDS-11454. Ranger integration for Docker Compose environment (apache…

55910a2

…#8575)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

HDDS-11454. Ranger integration for Docker Compose environment #8575

HDDS-11454. Ranger integration for Docker Compose environment #8575

Uh oh!

adoroszlai commented Jun 6, 2025

Uh oh!

adoroszlai commented Jun 6, 2025

Uh oh!

kumaab Jun 6, 2025

Uh oh!

kumaab commented Jun 6, 2025 •

edited

Loading

Uh oh!

adoroszlai commented Jun 13, 2025

Uh oh!

dombizita left a comment

Uh oh!

Uh oh!

adoroszlai commented Jun 16, 2025

Uh oh!

yazynin commented Jun 16, 2025 •

edited

Loading

Uh oh!

dombizita commented Jun 27, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

HDDS-11454. Ranger integration for Docker Compose environment #8575

HDDS-11454. Ranger integration for Docker Compose environment #8575

Uh oh!

Conversation

adoroszlai commented Jun 6, 2025

What changes were proposed in this pull request?

How was this patch tested?

Uh oh!

adoroszlai commented Jun 6, 2025

Uh oh!

kumaab Jun 6, 2025

Choose a reason for hiding this comment

Uh oh!

kumaab commented Jun 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

adoroszlai commented Jun 13, 2025

Uh oh!

dombizita left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

adoroszlai commented Jun 16, 2025

Uh oh!

yazynin commented Jun 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dombizita commented Jun 27, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

kumaab commented Jun 6, 2025 •

edited

Loading

yazynin commented Jun 16, 2025 •

edited

Loading