Skip to content

Conversation

@jojochuang
Copy link
Contributor

What changes were proposed in this pull request?

HDDS-13148. [Docs] Update Transparent Data Encryption doc.

Please describe your PR in detail:

  • Generated-by: Google Gemini 2.5 Pro (Preview) with the following prompt:
I want to update the current Ozone's Transparent Data Encryption
page https://ozone.apache.org/docs/edge/security/securingtde.html with the following instructions:

The Ozone TDE doc is written with the assumption that user is familiar with HDFS TDE, which may not be the case.

We should update the doc such that

(1) It does not require prior knowledge in HDFS TDE.

(2) Ozone can work with Hadoop KMS and Ranger KMS. We should mention Ranger KMS in the doc.

(3) For Ranger KMS, encryption key can also be managed by Ranger KMS management console or its REST API.

(4) hadoop key create enckey command has additional parameters: -size: specifies key bit length. Ozone supports 128 and 256 bits; -cipher: only AES/CTR/NoPadding (default) is supported as of now.

(5) Add reference to Transparent Encryption in HDFS: https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/TransparentEncryption.html and Hadoop KMS doc: https://hadoop.apache.org/docs/r3.4.1/hadoop-kms/index.html

(6) For the section Using Transparent Data Encryption from S3G, we should mention Ozone does not support S3-SSE (Server-Side Encryption) or S3-CSE (Client-Side Encryption). That said, Ozone S3 buckets can be encrypted using Ranger/Hadoop KMS to provide the same guarantee as S3-SSE with client-supplied key (S3 SSE-C).

(7) For section KMS Authorization: provide examples.

Be succinct. Insert new text to the existing content, instead of rewriting everything.

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-13148

How was this patch tested?

User doc only update.

Change-Id: I6b8e7c50063aedc7862f2b4ab3ecdebb44bfc38e
@jojochuang jojochuang requested a review from Copilot May 30, 2025 18:04
Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This PR updates the Transparent Data Encryption documentation to improve clarity and expand support instructions to include both Hadoop KMS and Ranger KMS. Key changes include:

  • Enhanced introduction and explanation of TDE and its configuration requirements
  • Updated instructions for creating encryption keys and encrypted buckets, including new command parameters
  • Revised S3 Gateway usage and Ranger KMS authorization examples
Comments suppressed due to low confidence (3)

hadoop-hdds/docs/content/security/SecuringTDE.md:156

  • Unintended text 'Youtube' appears in the permission list. Please replace or remove it to reflect the correct permission.
* Policy for `om` user (or the user running the Ozone Manager):
  * Resource: `keyname=enckey`
  * Permissions: `DECRYPT_EEK` (Decrypt Encrypted Encryption Key), `Youtube`

hadoop-hdds/docs/content/security/SecuringTDE.md:159

  • The permission list includes an unexpected 'Youtube' token. Please replace or remove it with the appropriate permission.
* Policy for S3 Gateway proxy user (e.g., `s3g_proxy`):
  * Resource: `keyname=enckey` (or specific keys for S3 buckets)
  * Permissions: `DECRYPT_EEK`, `Youtube`

hadoop-hdds/docs/content/security/SecuringTDE.md:162

  • The inclusion of 'Youtube' in the permission list seems accidental. It should be replaced or removed to correctly indicate the intended permissions.
* Policy for administrative users (e.g., `hdfs` or a keyadmin group):
  * Resource: `keyname=*` (or specific keys)
  * Permissions: `CREATE_KEY`, `DELETE_KEY`, `GET_KEYS`, `Youtube`, `ROLL_NEW_VERSION`

@jojochuang jojochuang requested review from myskov and smengcl May 30, 2025 18:06
Change-Id: Ia4c15cc0e04f31e63c8afd2b62c818fee66598d3
@jojochuang jojochuang requested a review from Copilot May 30, 2025 19:00
Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This PR updates the Ozone Transparent Data Encryption (TDE) documentation to make it more accessible for users without prior HDFS TDE knowledge and to clarify support for both Hadoop KMS and Ranger KMS. Key changes include:

  • Expanded explanation of TDE functionality and the configuration steps for a Key Management Server (KMS).
  • Detailed instructions for creating encryption keys and encrypted buckets, including new command parameters.
  • Revised guidance on using TDE with Ozone’s S3 Gateway and an updated section on KMS authorization with example policies.
Comments suppressed due to low confidence (1)

hadoop-hdds/docs/content/security/SecuringTDE.md:164

  • [nitpick] Ensure that resource identifiers in the Ranger KMS policy examples are consistently formatted (using code formatting where applicable) throughout the document for enhanced readability.
For example, when using Ranger KMS for authorization, you might have policies in Ranger KMS like:

@jojochuang jojochuang added the documentation Improvements or additions to documentation label May 30, 2025
Change-Id: I59521eb995ac4d4323f9ddf02fd19d767a496e9b
@jojochuang
Copy link
Contributor Author

Updated per Gemini's suggestion jojochuang#567

jojochuang and others added 4 commits June 4, 2025 09:49
Change-Id: Ifdc8ba98417966fea4d1f58fc01b8ccd5d8f8ce8
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
Change-Id: I49b3ff2f24eb0544945f9fc8839c62da29170590
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
@jojochuang jojochuang marked this pull request as ready for review June 4, 2025 17:20
Change-Id: Iee100e57085e7873d85d91254d7bc05c9c91cb1c
@jojochuang jojochuang requested a review from smengcl June 4, 2025 17:23
@smengcl smengcl merged commit 516bc96 into apache:master Jun 11, 2025
14 checks passed
@smengcl
Copy link
Contributor

smengcl commented Jun 11, 2025

Thanks @jojochuang for the doc improvement effort!

@jojochuang
Copy link
Contributor Author

Thanks @smengcl !

jojochuang added a commit to jojochuang/ozone that referenced this pull request Jun 13, 2025
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
sadanand48 pushed a commit to sadanand48/hadoop-ozone that referenced this pull request Jun 16, 2025
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
aswinshakil added a commit to aswinshakil/ozone that referenced this pull request Jun 20, 2025
…239-container-reconciliation

Commits: 62

da53b5b HDDS-13299. Fix failures related to delete (apache#8665)
8c1b439 HDDS-13296. Integration check always passes due to missing output (apache#8662)
7329859 HDDS-13023. Container checksum is missing after container import (apache#8459)
a0af93e HDDS-13292. Change `<? extends KeyValue>` to `<KeyValue>` in test (apache#8657)
f3050cf HDDS-13276. Use KEY_ONLY/VALUE_ONLY iterator in SCM/Datanode. (apache#8638)
e9c0a45 HDDS-13262. Simplify key name validation (apache#8619)
f713e57 HDDS-12482. Avoid using CommonConfigurationKeys (apache#8647)
b574709 HDDS-12924. datanode used space calculation optimization (apache#8365)
de683aa HDDS-13263. Refactor DB Checkpoint Utilities. (apache#8620)
97262aa HDDS-13256. Updated OM Snapshot Grafana Dashboard to reflect metric updates from HDDS-13181. (apache#8639)
9d2b415 HDDS-13234. Expired secret key can abort leader OM startup. (apache#8601)
d9049a2 HDDS-13220. Change Recon 'Negative usedBytes' message loglevel to DEBUG (apache#8648)
6df3077 HDDS-9223. Use protobuf for SnapshotDiffJobCodec (apache#8503)
a7fc290 HDDS-13236. Change Table methods not to throw IOException. (apache#8645)
9958f5b HDDS-13287. Upgrade commons-beanutils to 1.11.0 due to CVE-2025-48734 (apache#8646)
48aefea HDDS-13277. [Docs] Native C/C++ Ozone clients (apache#8630)
052d912 HDDS-13037. Let container create command support STANDALONE , RATIS and EC containers (apache#8559)
90ed60b HDDS-13279. Skip verifying Apache Ranger binaries in CI (apache#8633)
9bc53b2 HDDS-11513. All deletion configurations should be configurable without restart (apache#8003)
ac511ac HDDS-13259. Deletion Progress - Grafana Dashboard (apache#8617)
3370f42 HDDS-13246. Change `<? extend KeyValue>` to `<KeyValue>` in hadoop-hdds (apache#8631)
7af8c44 HDDS-11454. Ranger integration for Docker Compose environment (apache#8575)
5a3e4e7 HDDS-13273. Bump awssdk to 2.31.63 (apache#8626)
77138b8 HDDS-13254. Change table iterator to optionally read key or value. (apache#8621)
ce288b6 HDDS-13265. Simplify the page Access Ozone using HTTPFS REST API (apache#8629)
36fe888 HDDS-13275. Improve CheckNative implementation (apache#8628)
d38484e HDDS-13274. Bump sqlite-jdbc to 3.50.1.0 (apache#8627)
3f3ec43 HDDS-13266. `ozone debug checknative` to show OpenSSL lib (apache#8623)
8983a63 HDDS-13272. Bump junit to 5.13.1 (apache#8625)
a927113 HDDS-13271. [Docs] Minor text updates, reference links. (apache#8624)
7e77058 HDDS-13112. [Docs] OM Bootstrap can also happen when follower falls behind too much. (apache#8600)
fd13300 HDDS-10775. Support bucket ownership verification (apache#8558)
3ecf345 HDDS-13207. [Docs] Third party systems compatible with Ozone S3. (apache#8584)
ad5a507 HDDS-13035. SnapshotDeletingService should hold write locks while purging deleted snapshots (apache#8554)
38a9186 HDDS-12637. Increase max buffer size for tar entry read/write (apache#8618)
f31c264 HDDS-13045. Implement Immediate Triggering of Heartbeat when Volume Full (apache#8590)
0701d6a HDDS-13248. Remove `ozone debug replicas verify` option --output-dir (apache#8612)
ca1afe8 HDDS-13257. Remove separate split for shell integration tests (apache#8616)
5d6fe94 HDDS-13216. Standardize Container[Replica]NotFoundException messages (apache#8599)
1e47217 HDDS-13168. Fix error response format in CheckUploadContentTypeFilter (apache#8614)
6d4d423 HDDS-13181. Added metrics for internal Snapshot Operations. (apache#8606)
4a461b2 HDDS-10490. Intermittent NPE in TestSnapshotDiffManager#testLoadJobsOnStartUp (apache#8596)
bf29f7f HDDS-13235. The equals/hashCode methods in anonymous KeyValue classes may not work. (apache#8607)
6ff3ad6 HDDS-12873. Improve ContainerData statistics synchronization. (apache#8305)
09d3b27 HDDS-13244. TestSnapshotDeletingServiceIntegrationTest should close snapshots after deleting them (apache#8611)
931bc2d HDDS-13243. copy-rename-maven-plugin version is missing (apache#8605)
3b5985c HDDS-13244. Disable TestSnapshotDeletingServiceIntegrationTest
6bf009c HDDS-12927. metrics and log to indicate datanode crossing disk limits (apache#8573)
752da2b HDDS-12760. Intermittent Timeout in testImportedContainerIsClosed (apache#8349)
8c32363 HDDS-13050. Update StartFromDockerHub.md. (apache#8586)
ba1887c HDDS-13241. Fix some potential resource leaks (apache#8602)
bbaf71e HDDS-13130. Rename all instances of Disk Usage to Namespace usage (apache#8571)
0628386 HDDS-13142. Correct SCMPerformanceMetrics for delete operation. (apache#8592)
516bc96 HDDS-13148. [Docs] Update Transparent Data Encryption doc. (apache#8530)
5787135 HDDS-13229. [Doc] Fix incorrect CLI argument order in OM upgrade docs (apache#8598)
ba95074 HDDS-13107. Support limiting output of `ozone admin datanode list` (apache#8595)
e7f5544 HDDS-13171. Replace pipelineID if nodes are changed (apache#8562)
3c9d4d8 HDDS-13103. Correct transaction metrics in SCMBlockDeletingService. (apache#8516)
f62eb8a HDDS-13160. Remove SnapshotDirectoryCleaningService and refactor AbstractDeletingService (apache#8547)
b46e6b2 HDDS-13150. Fixed SnapshotLimitCheck when failures occur. (apache#8532)
203c1d3 HDDS-13206. Update documentation for Apache Ranger (apache#8583)
2072ef0 HDDS-13214. populate-cache fails due to unused dependency (apache#8594)

Conflicts:
	hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/impl/ContainerData.java
	hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/keyvalue/KeyValueContainer.java
	hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/keyvalue/helpers/KeyValueContainerUtil.java
	hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/keyvalue/statemachine/background/BlockDeletingTask.java
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Improvements or additions to documentation

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants